Color calibration of color image rendering devices

ABSTRACT

Color calibration of color image rendering devices, such as large color displays, which operate by either projection or emission of images, utilize internal color measurement instrument or external color measurement modules locatable on a wall or speaker. A dual use camera is provided for a portable or laptop computer, or a cellular phone, handset, personal digital assistant or other handheld device with a digital camera, in which one of the camera or a display is movable with respect to the other to enable the camera in a first mode to capture images of the display for enabling calibration of the display, and in a second mode for capturing image other than of the display. The displays may represent rendering devices for enabling virtual proofing in a network, or may be part of stand-alone systems and apparatuses for color calibration. Improved calibration is also provided for sensing and correcting for non-uniformities of rendering devices, such as color displays, printer, presses, or other color image rendering device.

This application is a continuation of U.S. patent application Ser. No.15/894,545, filed Feb. 12, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/368,022, filed Dec. 2, 2016, now U.S. Pat. No.9,894,340, which is a continuation of U.S. patent application Ser. No.14/933,781, filed Nov. 5, 2015, now U.S. Pat. No. 9,516,288, which is acontinuation of U.S. patent application Ser. No. 14/161,121, filed Jan.22, 2014, now abandoned, which is a divisional of U.S. patentapplication Ser. No. 12/799,930, filed May 5, 2010, now U.S. Pat. No.8,638,340, which is a divisional of U.S. patent application Ser. No.11/216,784, filed Aug. 31, 2005, now U.S. Pat. No. 7,728,845. U.S. Pat.Nos. 6,043,909, 6,157,735, 6,459,425, and 6,750,992 are hereinincorporated by reference.

FIELD OF THE INVENTION

The present invention relates to systems and apparatuses for colorcalibration of color image rendering devices, such as large colordisplays operating by either projection or emission of images, ordisplays associated with desktop, workstations, laptop, handsets, orhandheld computers. The displays of the present invention may representa rendering device as described in the above-incorporated patents forenabling virtual proofing in a network, or may be part of stand-alonesystems and apparatuses for color calibration. In addition tocalibration as described in the above incorporated patents, improvedcalibration is described for sensing and correcting for non-uniformitiesof rendering devices, such as color displays, printer, presses, or othercolor image rendering device.

BACKGROUND OF THE INVENTION

Color display calibration is described in U.S. Pat. Nos. 6,043,909,6,157,735, 6,459,425 and 6,750,992, in which sensor configurations arepresented for different display types. One such type of arrangement issuited to a projection display as described, for example, at column 16,lines 31-35, of U.S. Pat. No. 6,043,909. Such sensors were located nearthe source of projected light and were as close as possible to the lineof sight. However, many contemporary displays used in large screenapplications, such as home theatre or commercial projectors, may benefitfrom color calibration to assure color accuracy, and further mayrepresent rendering devices useful for virtual proofing described in theabove U.S. patents.

Further, often color display are not uniform over their field of view,and thus areas of such display may be appear brighter than others, andmay appear to have different color balance amongst different colorchannels at different points. Thus, it would also be useful to detectnon-uniformities in color displays and correct for such non-uniformity,thereby providing consistency in displayed color. Furthermore, it wouldbe useful if such detection and correction of non-uniformity may beextended to other rendering devices, such as printers and presses.

SUMMARY OF THE INVENTION

It is one object of the present invention to provide color displaycalibration systems for large screen displays which operate by eitherprojection or non-projection (e.g., self-luminous or LCD).

Another object of the invention is to provide a dual use camera for aportable device in which one of the camera or a display of the portabledevice is movable with respect to the other to enable the camera in afirst mode to capture images of the display for enabling displaycalibration, and in a second mode for capturing image other than of thedisplay, such as of the user for video teleconferencing or for enablingcalibration of other color rendering devices, such as color displays,printers, or presses.

It is a further object of the present invention to detect and correctnon-uniformities of color displays and other rendering devices, such asprinters and presses.

Briefly described, the present invention embodies a color projectiondisplay device for outputting images onto a screen having a colormeasuring instrument within the housing of the display device, where thecolor measurement instrument utilizes either the projector's optics, orseparate optics for imaging reflected light from the screen. Theelectronics of the display device provides for calibration of thedisplay utilizing measurement data from the color measurementinstrument, and/or provides for virtual proofing with one or more otherrendering devices over a network.

When such color measurement instrument is not integrated in suchhousing, a color measurement module may be provided which is attachableto either the wall or a speaker opposite the screen, which is especiallyuseful in home or commercial theater environments. A control unit, whichcommunicates image data for output by the display device, or theelectronics of the display device enables calibration of the displayand/or virtual proofing in accordance with color measurement datareceived from the color measurement module. Such color measurementmodule may also be utilized for other large screen display devices, suchas rear projection color display device or a self-luminous (“emissive”)color display, based on technologies, such as LCD, plasma, OLED, and thelike, to enable calibration of the display and/or virtual proofing.Methods are provided for manually or automatically focusing and aligninga sensor of the color measuring module with a reference image on thescreen, and for detecting and correcting for spatial luminousdistortions detected between the reference image and the sensed image.

The present invention also provides automatic color calibration for aportable device with a color display and an attached or built-in colorcamera. A processor of the portable device calibrates color of thedisplay in accordance with one or more images of the display captured bythe camera. One of the camera or the display is movable with respect tothe other to enable the camera to capture one or more images of saiddisplay to enable calibration, and one of the camera or display beingmovable with respect to the other to capture images other than of thedisplay. As such the camera of the portable device is a dual use camera,since in a first mode it captures one or more images of the display toenable calibration, and in a second mode captures images other than ofthe display (such as of the user in the case of video teleconferencing).In the first mode, the camera is automatically aligned with the screenfor carrying out color calibration, spatial uniformity correction,and/or virtual proofing by software or program operating on the portabledevice. The camera may also have a third use of being oriented forimaging another color display of another image display device (e.g.,theater display system) to provide color image data representative ofthe another color display, and one of the processor of the portabledevice or the another image display device enables calibration (and/orspatial uniformity correction or virtual proofing) of the another colordisplay in accordance with the color image data. Further, such third useof the portable device may be applied to other color rendering devicesthan color displays, such as printers, presses, and copiers, whichoutput color image data onto a surface, such as media, in which theprocessor of the portable device or the other color rendering deviceenables calibration (and/or spatial uniformity correction or virtualproofing) of the color rendering device in accordance with the colorimage data.

The processor of the portable device calibrates color of the display inaccordance with color image data captured by said camera representingcolor of one or more calibration forms outputted by the processor on thedisplay. The camera may be further capable of capturing images providinginformation regarding spatial non-uniformity of color reproductionacross the screen of the display.

The portable device may be a portable computer, and an arm is providedhaving a first end attached to the frame of the display and a second endattached to the camera, in which the camera is pivotally mounted at thesecond end to enable the camera to be oriented in a first mode tocapture one or more images of the display to enable calibration, and ina second mode to capture images other than of the display. The portabledevice may also be a cellular phone or other handset with a built-incamera having an upper portion with the display and a lower portion withthe camera. The upper portion is pivotally mounted with respect to thelower portion to enable the upper portion to be pivoted in a first modeto capture one or more images of the display for display calibration,and in a second mode to capture images other than of the display. Theportable device may also represent a cellular telephone or handsetwithout any pivotal portions, where its camera is attached to thecellular phone by an arm that is pivotal in one or more dimensions toorient the camera in a first mode to capture one or more images of thedisplay for display calibration, or in a second mode to capture imageother than of the display.

In the case of the portable device representing a cellular phone orhandset, the camera may also be capable of being oriented to capture animage representative of the amount of ambient illumination, such thatthe backlight of the display is turned on or off in accordance with theamount of ambient illumination exceeding a threshold, or a gradientfunction, or turned on or off in accordance with the amount of ambientillumination exceeding a threshold and turned off after an intervalexpires since last input via the user interface of the cellular phone orhandset.

Apparatus, systems and methods are also provided by the presentinvention for detecting and correcting for spatial non-uniformity of adisplay screen of large screen display devices, portable or laptopcomputer, color monitors (e.g., CRT or LCD), or the like, or hardcopycolor image rendering devices, such as printers or presses, or smallerportable devices, such as cellular phones, handset, or personal dataassistants with a camera. A color measuring instrument, such as animaging sensor (e.g., CCD) measures rendered flat field images for eachof the color channels, and for each measured image determines the pixelcoordinate (x,y) that has the lowest fraction (or ratio) Imin/Imax ofmeasured intensity, where Imin and Imax are the measured lowestintensity pixel and highest intensity pixel, respectively, in a channel.Using first the channel and its pixel (x,y) having the lowest fractionoverall, all of the other pixels in each of the other channels arereduced in intensity while providing the desired neutral or whitebalance (i.e., white point) of pixel (x,y). If such intensity reductionin the other channels for pixel (x,y) is greater than Imin/Imax for themeasured flat field image for the other channel, than another pixel isselected in such other channels, and then all of the other pixels ineach of the channels are reduced in intensity while providing thedesired neutral or white balance (i.e., white point) for that pixel. Ifno such white balance can be determined for that pixel, one pixel isfound with or given the correct white balance, and the other pixelcoordinates in each channel are each incrementally reduced in intensityin one or more channels until each pixel has a proper neutral balance.The resulting reduction in intensity for each pixel coordinate in eachchannel are stored in a correction non-uniformities table. Such table isapplied to image data prior to being outputted by the rendering device.

An apparatus and method are also provided by the present invention forapplying spatial uniformity correction in a production printing context,in which positional dependences of color are accounted for on presssheets. Corrections for positional dependencies of pixels rendered by apress may be modified by one or more of spatial uniformity correctiontable, a rendering color transformation, or a color-to-color'transformation within the signature of the press (i.e., the relativeprint densities achievable at a trailing location of an impression as afunction of reproduction densities required at a more leading positionwithin the same impression (transfer of ink to paper)).

The apparatuses, systems, and methods described herein may be used inthe virtual proof environment described in the above incorporated U.S.patents, but are not limited for use in such environments, and mayprovide stand-alone calibration of a rendering device by software orprogram, operative in a control unit or host computer in communicationwith such rendering device and color measuring instrument, for colorcalibration of color image rendering systems to assure that color isaccurately displayed with respect to rendered images as measured by suchcolor measurement instrument.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features, objects, and advantages of theinvention will become more apparent from a reading of the followingdetailed description in connection with the accompanying drawings, inwhich:

FIG. 1 shows the system in accordance with the present invention;

FIG. 2 shows a configuration of a color measuring instrument sensor fora video screen display;

FIG. 3A shows geometries of a color measuring instrument sensor formaking non-contact color measurements of reflective substrate;

FIG. 3B shows use of a color measurement instrument to estimate acomposite spectral function given knowledge of the underlying spectralfunctions of the colorants being mixed;

FIG. 4A illustrates the devices of the system separated into deviceclasses inheriting various structures and procedures for colorcalibration and transformation;

FIG. 4B is a process diagram for color transformation of a class ofdevices including linear color measuring instruments;

FIG. 4C is a process diagram for color transformation of a class ofrendering devices including video-color displays;

FIG. 5 is a process diagram for calibrating a class of rendering devicesincluding printers and presses at a node in the system of FIG. 1 toprovide color transformation information;

FIG. 6A is a flow chart detailing step 1 in FIG. 5, preparinglinearization functions;

FIG. 6B is a flow chart detailing step 2 in FIG. 5, renderingcalibration forms;

FIG. 7 is a flow chart detailing step 3 in FIG. 5, measuring calibrationforms and providing calibration data;

FIG. 8 is a flow chart detailing step 4 in FIG. 5, building a forwardmodel based on the calibration data from step 3 of FIG. 5;

FIG. 9A is a flow chart detailing step 5 in FIG. 5, preparing gamutdescriptor data for the rendering device and preparing a forward modeltable based on the forward model from step 4 in FIG. 5;

FIG. 9B is an illustration of the operators and operands evaluating thepolynomial function of the forward model in the case of two independent(colorant) variables, C and M;

FIG. 9C depicts a hypercube in the coordinates of the Cyan, Magenta,Yellow and Black colorants in which all colors producible with a CMYKprinter are contained within the hypercube;

FIG. 9D is an illustration of a data structure for interpolation in 3dimensions which may use either pre- or post-conditioning look-uptables;

FIG. 9E is a graphical illustration of linear interpolator in twodimensions;

FIG. 10A is a flow chart detailing step 6 in FIG. 5, inverting theforward model table to provide a prototype transformation table;

FIG. 10B is an example of the hypercube of FIG. 9C where the colorantvalues are transformed into device independent color coordinates;

FIG. 11 is a flow chart detailing step 7 of FIG. 5, finishing the gamutdescriptor data;

FIG. 12 is a flow chart detailing step 8 of FIG. 5, filling in anymissing entries of prototype transformation table, computing blackutilization (GCR) functions for all colors of the table having multipleblack solutions and marking unprintable colors NULL;

FIG. 13 is a flow chart detailing step 9 of FIG. 5 which includes:converting colorants of prototype transformation table based on blackcolor data; building color to color' transform table based on gamutconfiguration data; and combining the color to color' transformationtable and the converted prototype transformation table to provide arendering table;

FIG. 14 is an illustration of the construction of a simple gamutoperator of the gamut configuration data embodying properties ofinvertibility and reciprocality;

FIGS. 15A and 15B show the constituents of local and shareablecomponents in the data structure of the Virtual Proof;

FIG. 15C is an example of a tagged file format for the shared componentsof the Virtual Proof of FIGS. 15A and 15B;

FIG. 16A is a flow chart of the process for calibrating a renderingdevice having more than four colorants by adding non-neutral auxiliarycolorants to a rendering transformation;

FIG. 16B is a flow chart of the process for calibrating a renderingdevice having more than four colorants by adding a neutral auxiliarycolorant to a rendering transformation, in which FIGS. 16A and 16B areshown connected;

FIG. 17 is a flow chart showing the process for preparing a gamutfilter, a data structure which facilitates the comparison of gamuts oftwo or more rendering devices;

FIGS. 18A and 18B are a flow chart for virtual proofing using the systemof the present invention;

FIG. 19 is a flow chart of the verification procedures employed toestimate process variations based on measurements of color errors;

FIG. 20 is a flow chart of the process of preparing a three-dimensionalcolor histogram of color image data in providing aresolution-independent analysis of color error relative to a criterion;

FIG. 21A is a menu of the Graphical User Interface (GUI) to theapplication software to enable configuration of the network of nodes,remote conferencing and proofing and oversight of the processes involvedin calibrating devices in the systems of FIG. 1;

FIG. 21B is a menu at the second level of hierarchy in the GUI toprovide access to tools for configuring the network connections andcommunications protocols;

FIG. 21C is a menu at the second level of hierarchy in the GUI to enablea user to manipulate the process of making color transformations at arendering device;

FIG. 21D is a menu at the second level of hierarchy in the GUI to enablea user to oversee the process of using color transformations at arendering device;

FIG. 21E is a menu at the third level of hierarchy of the GUI whichdepicts the User interface to black utilization tools in providing blackcolor data, and to neutral colorant definitions;

FIG. 21F is a menu at the third level of hierarchy of the GUI whichdepicts the User interface to gamut processing at a rendering device inthe network;

FIG. 22 is a block diagram of an example of the system of FIG. 1;

FIG. 23 is a graph illustrating the spectrum of the green phosphoremission from a cathode ray tube (CRT) color display;

FIG. 23A is a graph illustrating the spectral emission of a red phosphoremission as recorded by two different models of colorimeters;

FIG. 24 is a block diagram of the configuration of a color measuringinstrument for a video screen display which is similar to FIG. 2.

FIG. 24A is a perspective view showing an example of an assembly formounting a color measuring instrument to a video screen display;

FIGS. 24B, 24C and 24D are perspective view of the cowel, arm member,and one of the brackets, respectively, of FIG. 24A;

FIG. 25 is a block diagram of the configuration of a color measuringinstrument for a video screen display of FIG. 24 which includes aviewing box;

FIG. 26 is a block diagram showing the viewing box of FIG. 25 in furtherdetail;

FIG. 27 is a block diagram of part of the sensor of the colormeasurement instrument located in the cowel of the assembly of FIG. 24A;

FIG. 28 is a block diagram of the color measurement instrument with thesensor of FIG. 27 and control circuitry;

FIG. 29 is a table of the command set used by a computer to communicatewith the color measurement instrument of FIG. 28;

FIG. 30 is a graph of the gamma of the green color channel of the colormeasurement instrument of FIG. 28;

FIG. 31 is a high level flow chart showing the operation of the systemin accordance with the present invention for soft proofing to a colordisplay;

FIG. 32 is a flow chart for the process of maintaining a color displayin calibration;

FIG. 33 is a block diagram of a color measurement instrument inaccordance with the present invention utilizing a dual beamspectrograph;

FIG. 34 is a block diagram of the a color measurement instrument inaccordance with the present invention utilizing a concentricspectrograph;

FIG. 35 is a block diagram of a front projection display device withcolor calibration in accordance with the present invention in whichprojection optics project upon a surface and the same optics are used toreceive reflected light from the surface onto a color sensor;

FIG. 36 is a block diagram similar to FIG. 35 in which different opticsthan the projection optics receive reflected light onto the colorsensor;

FIG. 37 is perspective view of a home theatre system in a room having aflat panel display and multiple speakers about the walls of the room inwhich a color measuring module is attached or integrated with a speakerof the home theatre system to provide color calibration of the flatpanel display;

FIG. 37A is a block diagram of the color measuring module of FIG. 37;

FIG. 37B is similar to FIG. 37 with a front color projection displaydevice and a color measuring module attached or integrated with aspeaker to provide color calibration of the projection display device;

FIG. 38 is a flow chart for the process of focusing and aligning thecolor measure module of FIGS. 37, 37A, and 37B and determining a spatialluminance distortion function;

FIG. 38A is a flow chart of the Perform Autofocus block of FIG. 38;

FIG. 39 is a perspective view of a portable, laptop, or notebookcomputer with an embedded, pivotable arm for mounting a camera capableof supporting both teleconferencing and calibration functionalities;

FIGS. 39A and 39B are top and side views, respectively, of the camera ofFIG. 39;

FIG. 39C is a plan view of a portable device of a cellular phone havinga built-in color camera utilizing color calibration in accordance withthe present invention;

FIG. 39D is a side view of the portable device of FIG. 39C;

FIG. 39E is a block diagram of the electronics of the portable device ofFIG. 39C;

FIG. 39F illustrates the structure of menus on the user interface of theportable device of FIG. 39C enabling the user to select camerafunctions;

FIG. 39G is a flowchart for the process of automatically controlling thebacklight of the portable device of FIG. 39C;

FIG. 39H is a graph illustrating an example of gradient function fordetermining the level of backlight illumination for the display of aportable device of FIG. 39C for different levels of ambient illuminationmeasured by the device;

FIG. 39I is a flowchart for the process of calibrating another colordisplay device (or system) using the portable device of FIG. 39C;

FIG. 39J is a flowchart for the process of characterizing the responsesof the portable device of FIG. 39C in terms of standard color units;

FIG. 39K is a block diagram of a support or base which may be coupled tothe portable device of FIG. 39C when calibrating another color displaysystem;

FIG. 39L is a diagram of an attachment for mounting the portable deviceof FIG. 39C or other digital camera in place of the camera of FIGS. 39,39A and 39B, such that the portable device or other digital camera iscoupled to the computer of FIG. 39 and the camera of the portable devicecan provide the same functionality as the camera of FIGS. 39, 39A and39B;

FIG. 39M is a broken partial view of the handset of FIG. 39C to show anoptional magnifier attached to the handset when utilizing a miniaturehigh resolution display;

FIG. 39N is a broken side view of FIG. 39M;

FIG. 40 illustrates an example of a display surface that is non-uniformin response to a flat field input, in which points A, B and C shouldhave the same color, but do not.

FIG. 41 is a block diagram of the flow of data through a system thatdrives a display having spatial non-uniformity correction;

FIG. 42 is a block diagram of an apparatus for flattening the displaysurface by providing spatial non-uniformity correction;

FIGS. 43A and 43B are connected flow charts of the process in theapparatus of FIG. 42 for detecting spatial non-uniformity and forpreparing a table for correction thereof while preserving desiredneutral or white balance;

FIG. 44 is a block diagram of a system and apparatus for applyingspatial uniformity correction of a printable display surface in aproduction printing (press) context;

FIG. 45 is a flow diagram for the method of preparing and distributingtransformations for color image data intended to enable proofing ofpress performance and to improve color image reproduction by the pressof FIG. 44; and

FIG. 46 is a block diagram of workstations enabling spatialnon-uniformity correction in an application involving a web offset pressfor publication printing and the interaction with pre-press workflow,especially at the imposition stage.

DETAILED DESCRIPTION OF THE INVENTION

The color calibration of the color video displays, described later inconnection with FIGS. 35-38, may be part of a virtual proofingenvironment described in the incorporated patents. As such, thefollowing describes the system enabling such environment as set forth inthe incorporated U.S. Pat. No. 6,459,425.

Referring to FIG. 1, the system 100 of the present invention is shown.System 100 has a network 11 having a pipe 11 a through which multiplenodes (or sites) of network 11 can be linked for data flow betweennodes. Network 11 may be a telecommunication network, WAN, LAN (with aserver) or Internet based. Two types of nodes are present in system 100,prototype nodes 102 and production nodes 104. For purposes ofillustration, only a general node of each type is shown in FIG. 1,however there may be multiple nodes of each type in network 11. Network11 is modifiable to be configured by any one node to connect any two ormore nodes in system 100. Each node has a micro-processor basedcomputer, with a network communication device, such as a modem, which ispart of a system having a rendering device for producing colorreproduction and color measuring instrument (CMI) for measuring thecolor output of the rendering device. The computer may be a programmablegeneral purpose computer or mainframe computer. Although a computer at anode is preferred, alternatively, the computer may be omitted at a nodeand the node operated remotely via pipe 11 a from another node.

Prototype nodes 102 allow a user to perform pre-publishing functions insystem 100, such as proofing (hard or soft), as well as the input ofdigital color image data. A user may interface with the node throughstandard interface devices, such as a keyboard or mouse. Renderingdevices in system 100 define any type of system or device for presentinga color reproduction in response to digital color signals. The renderingdevices of prototype node 102 are proofing devices, such as video screendisplay device 17 or proofer device 16. Proofing device 16 are hard copydevices, such as analog film-based devices, dye diffusion thermaltransfer devices, ink jet printers, xerographic printers, and othersimilar devices. Video screen display 17 is useful for soft proofing(without a hard copy) and may be a high resolution video projectiondisplay for projecting images onto paper substrate(s) to be used involume reproduction with resolution sufficient to reveal moiré, i.e.,halftone frequency beating patterns. Note that proofing devices aretypically used to represent the performance of a client, such as aproduction rendering device described below at production node 104. TheCMI associated with each proofing device is referred to as a standardobserver meter (SOM) 13 and provides color measurement data from imagesfrom the proofing device. SOMs 13 have the capability of measuring coloras humans see it using the internationally accepted Standard Observermentioned earlier, and will be described in more detail later.

One of the pre-publishing functions supported by prototype node 102 isdesigning page layouts. Accordingly, a user or designer at nodes 102 caninput digital color graphical/image data from a storage 19, which may bea hard drive, or from other sources. The color image data may consist ofpage layouts having images at low and high resolutions, such as RGB. Auser at the node can define color preferences for rendering of the colorimage data, and later modify such preferences. The rendering of theinputted color image data at rendering devices to create soft or hardproofs is discussed later.

Production nodes 104 of network 11 control a production rendering devicevia the device's control system. Production rendering devices includevolume production machinery, such as press 15, which includes gravurepresses, offset presses, electrophotographic printing machines, ink jetprinters, flexographic presses, and the like. In addition, productionnodes 104 may also have one or more rendering devices and SOMs 13 of aprototype node 102, such as proofing devices 20, which allows proofingto occur at a production site. CMIs of node 104 are called imagicals.Like SOMs 13 of prototype nodes 102, imagicals 14 provide color data forimages rendered by press 15 in color coordinates of the StandardObserver. Proofing devices at prototype nodes 102 may also be outfittedwith imagicals 14 which may incorporate a SOM 13. The differencesbetween SOMs 13 and imagicals 14 will be described later. The maindistinctions between production rendering devices and proofing devicesare that the proofing devices are typically called upon to representsome other device, referred to herein as a client, such as printingpress 15 of a production node 104 and such presses may have interfacesto and mechanisms of control that are different from those of proofers.

At a production rendering device, the circuitry at node 104 differs fromnode 102 because it interfaces with inking control systems of theproduction rendering device to maintain its color quality during volumereproduction. This allows control of the actual marking process,including variables such as ink film thickness, toner density, and thelike, by supporting on-line colorimetry from a CMI within image areas ofprinted sheets. Analysis of the CMI data can be used to produce errorsignals in CIELAB color difference units or in other units suitable forinterface to commercially available inking control systems.

The nodes 102 and 104 each provide circuitry, which includes thecomputer and modem described above, for computation and communication.This circuitry operates responsive to programming of interface andapplication software stored at the nodes 102 and 104, and received usercommands at the nodes. The processes defined by such programming operatesystem 100. The circuitry perform several functions. First, it acceptsmeasurement data from CMIs and computes color transformation functionsto translate between human-perceptible colors of the measurement datainto rendering device colorant values. Second, it processes andtransmits color graphical/image data from one node or site in a network11 to another. Third, it can issue reading instructions to CMIs mountedon a rendering device to measure rendered color images, and issuerendering instructions to a rendering device at the node using a storedcolor transformation. Fourth, the circuitry performs communications insystem 100 in accordance with protocols for local or wide area networks,or telecommunications networks based on modem (either direct or mediatedby Internet connection—note that Internet connectivity is not limited tomodem,) satellite link, T1 or similar leased line technologies, ISDN,SMDS and related switched linkages, including Asynchronous TransferMode-enabled versions, TCP/IP, token ring, and the like. Fifth, thecircuitry implements calibration of rendering devices to a common, humanperceptible language of color, such as CIE, defined earlier, byproducing and storing color transformation information. Sixth, thecircuitry performs verification of the calibration of the renderingdevice to maintain accuracy of the stored color transformationinformation. These and other capabilities of the circuitry at a nodewill become apparent from the below discussion which describe furtherthe processes referred to above.

A feature of the present invention is a data structure operating withinsystem 100 called a Virtual Proof, hereafter called VP 12. The VP datastructure is a file structure for storing and transmitting filesrepresenting color transformation information between network 11 nodes.The contents of these files is outlined later. The VP is dynamic becauseit can be revised by nodes to assure the output color (colorants) of arendering device using data from CMIs. Preferably, the VP does notcontain color image data representing page layouts, and is associatedwith the page layouts. However, it can alternatively have files storingsome image data, although being separable from the often bulky highresolution image data representing the page layouts. The VP hascomponents or files shared by the nodes in network 11, and localcomponents or files present only at each node. Shared components arethose useful by more than one node in network 11, while local componentsare particular to information of each individual node's renderingdevice. Shared components are transmitted by the circuitry of each nodeto other nodes of network 11 via pipe 11 a. Preferably, VP sharedcomponents are compact for transmission from node to node in network 11quickly. These shared VP components include files representing the usercolor preferences inputted at node 102 or 104, which is needed by eachnode in calibrating its rendering device. Each rendering device has itsown version of a VP stored at its associated node which represents theshared VP components and local components for that particular renderingdevice. In FIG. 1, VP₁ VP₂, VP₃ and VP₄ represent the versions of thevirtual proof for each rendering device. The arrows from SOM 13 orimagical 14 represents the measurement data received by the nodeincorporated into color calibration data, which is stored in a localcomponent of the VP.

The VP provides system 100 with many useful features, which includeremote proofing for both intermediate and final approval of colorproducts, conferencing at multiple nodes in the network between userswhich may have different rendering devices, and distributing colorpreference data with or without page layout image data. The conferencingmentioned above allows users to negotiate over the colors appearing inpage layout and to confer about color corrections. For example,conferencing may use video displays 17 (soft proofs) of the page layoutsusing remote annotation software, such as imagexpo. Another importantfeature of the VP is that it is modifiable such that as changes occur ata rendering device, such as in inks or substrates, system 100 canautomatically adjust the rendering device's calibration. In addition,adjustments to calibration may be performed on demand by a user. Thisallows a user to update color preferences, such as color assignments ofpage layouts being rendered by rendering devices in the network withoutretransmitting the entirety of the image data.

A feature of system 100 is that it compensates for differences in thegamuts of different devices. As described earlier, a gamut is the subsetof humanly perceivable colors that may be captured or rendered by adevice. The preceding definition implies that the ideal or limitinggamut is the set of all colors that a normal human can see. It isimportant to distinguish between receptive and rendering gamuts. Theformer refers to the gamut of a sensor or camera of a CMI, or human. Thelatter refers to the colors that an output rendering device is capableof producing in a medium by application of its colorants. Although itmay be possible to design a rendering device that may produce all thecolors we can see under suitable circumstances, rendering gamuts aregenerally smaller than the human perceptual gamut due to the propertiesand limitations of practical reproduction media. For example, the gamutof a color print viewed in reflected light is generally smaller thanthat of a video display device viewed in controlled illumination whichis generally smaller than the gamut available with positive photographictransparencies. All the foregoing rendering gamuts are generally smallerthan receptive gamuts.

The CMI's of FIGS. 2 and 3A are colorimeters such as discrete (unitary)colorimeters (SOM 13) or imaging colorimeters (imagical 14) which may beused in conjunction with single-channel light-detectors. Thesecolorimeters may be stand-alone units or built-in to a rendering device.As stated earlier, the CMIs are controlled by their associated nodes insystem 100 for calibration and verification of rendering devices. SOMs13 and imagicals 14 are specialized to measure color in different ways.SOMs are suited for measuring a spatially homogeneous patch of color,preferably in a multiplicity of spectral bands. Preferably, at least 15spectral bands spanning the visible spectrum are sampled, making a SOMmore-than-3-channel input device. The transformation from relativespectral energy or reflectance to image colors employs the convolution.An example of a SOM is a unitary colorimeter or a spectrophotometer of aU.S. Pat. No. 5,319,437 issued to Van Aken et al.

However, the distinction between SOMs and imagicals is not intrinsic. ASOM with a sufficiently restricted aperture and area of view and whichcould perform the spectral integrations sufficiently rapidly and whichcould scan rasters of image data may qualify as an imagical. Imagicalsare suited for multi-scale (i.e., variable resolution) imagingcolorimetry, consisting of an array of photosensors, such as CCDs,capable of sensing color, as would the Standard Observer.

SOM 13 is calibrated against a reference standard illumination sourcewhose calibration is traceable to the U. S. National Institute ofStandards and Technology or similar organization. The calibration of SOM13 is generally set by the factory producing the device. SOM 13 shouldbe periodically recalibrated to assure its reliability. Calibration ofimagicals will be described later.

Further, SOM 13 may be used in conjunction with an imagical. SOM 13 canprovide a check on the calibration of imagical 14 by sampling some ofthe same colors measured by the imagical and providing reference data tocompare against the imagical measurements. Under suitable circumstancesthe SOM enables a spectral interpretation of what is seen by theimagical so that a spectral illumination function characteristic ofviewing can be substituted for that used in measurement as described inconnection with FIG. 3B.

Referring to FIGS. 2 and 3A, preferred configurations for sensors forCMIs in system 100 are shown. Because colorimetric accuracy of CMIs andtheir ease-of-use are desirable, the preferred configuration of CMIdevice colorimetric sensors is to have them be as unobtrusive aspossible to the user. Thus, in the preferred embodiment, CMIs arebuilt-in to the rendering device.

In the case of a conventional video display or monitor, FIG. 2 shows thepreferred embodiment for sensor of a CMI. A cowel 26 attaches to a upperchassis 27(a) to frame a faceplate or screen 24 of video display 22 andto shield the display from most ambient illumination. A fiber opticpickup (not shown) coupled to a projection type lens or lens system 28plugs into cowel 26 in order to measure color of screen 24 withoutactually touching the screen or requiring placement by the user. Path 30shows that the line of sight of the lens and fiber optic pickup reflectsoff faceplate 24 and views the blackened inner surface of a lower cowel32 attached to lower chassis 27(b) such that it does not see specularlyreflected light reflected from faceplate 24. Nonetheless, operation ofdisplay 22 is preferably in an environment with subdued illumination.

Preferably the CMI for a display screen is as a unitary colorimeter SOM13. The unitary colorimeter takes color measurements via lens system 28when needed in response to instructions from circuitry at a node.Unitary colorimeter SOM 13 can measure relatively small areas of screen24 using a sensor connected to the fiber optic pickup. This sensor canbe a spectral sensor, a 3 or 4 filter colorimeter or a single channelsensor. A spectral sensor must be able to resolve 2 nanometer wavebandsacross at least the long wave (red) end of the spectrum in order to makean adequate measurement of the red phosphor of conventional CRTs. Thiscould be accomplished with a grating monochromator and linear photodiodearray, or linearly variable interference filter and diode array.Scanning the red end of the spectrum may be performed with a narrowly,electronically-tuned, spectrally variable optical filter in order tolocate red phosphor peaks exactly. If a 3 or 4 channel filtercolorimeter is used, compatibility with system 100 requires that thespectral sensitivity of the arrangement be a linear combination of thehuman color matching functions with acceptable precision. The videodisplay 22 phosphors change on a slow time scale. Therefore, providedthat the primary chromaticities of the display are measured on a regularschedule, the SOM of FIG. 2 may be replaced by a single-channel meterfor routine measurements of gamma (the voltage in-photons outcharacteristic of the device) very accurately for the three channels.

Alternatively, an imagical may be used instead of a unitary colorimeterSOM 13 in FIG. 2. In this case the sensor of the imagical may becentered in a door (not shown) which is attached to cover the aperture29 of cowel 26 and 32 (the cowel may surround the entire perimeter ofthe chassis) so that the sensor views faceplate 24 center along a lineof sight. Imagical may acquire data needed to flatten the screen, i.e.,make it regionally homogeneous in light output or to compensate forinteractions among adjacent pixels in complex imagery. However, both ofthese factors are second-order effects.

It was noted earlier that the preferred embodiment utilizes a videodisplay 17 that projects the image, preferably onto printing paper. Inthis configuration, the CMI which monitors output is situated near thesource of projected light, as close as possible to being on a line ofsight. Preferably, a projection video display is capable of resolutionsexceeding 1200 lines per inch. In this way, it is possible to simulaterosettes, moires and details of image structure on the hard copy and toshow the actual structure of images captured from prints with a CMIequipped with macro optics. To provide high resolution projection,butting together a composite image using several limited-area-displaysmay be performed. Regardless of type of display 17, the display may useadditional primaries in order to extend gamut or better simulatesubtractive colorants.

FIG. 3A shows an example of a sensor of a CMI for measuring a substrate34 (hard proof), such as produced by a proofer device. In the case of adigital proofing device, such as a dye sublimation or ink jet printer,it is desirable to position a dual fiber optic pickup/illuminator 36arrayed for 45 and 90 degrees measurement geometry near the print headin order to sample recently printed color pixels of substrate 34.Pickup/illuminator 36 have projection-type lenses coupled to both asensor fiber and a illumination fiber within a light shielding sleeve toimplement non-contact measurements and to protect against stray light.Pickup 36 relays light to an analysis module of a CMI in which,preferably, a spectral colorimetric measurement is made. If placementnear, the print head is not practical, then placement over the exit pathof printed sheets is preferred. This sort of placement is preferred forproofing devices which are page printers, such as electrophotographic orconventional, analog proofers. As was the case with monitors, the use ofa single-channel device to monitor linearization is compatible, providedthat the nature of the device variation over time is compatible withintermittent, full, colorimetric calibration. For example, it may besufficient to calibrate a dye sublimation proofer only once for eachchange of ribbon. Although it is preferred to build the CMI into theproofing device in order to minimize user involvement with the process,system 100 can alternatively require the user to place the printed copyon a X-Y stage equipped with the dual fiber optic pickup/illuminator 36.

At production node 104 of system 100, it is preferred that imagesrendered from a press are captured as soon as possible after they areprinted, i.e., after all the colorants are applied. In this way the datanecessary for control are available promptly. Although system 100 mayestimate the color-error of printed sheets relative to an aim value, thepreferred embodiment applies imaging colorimetry of a CMI to theanalysis of the image area. Preferably, there is a spectral component tothe CMI measurement such that interactions of the colorants with thestock or printing substrate can be analyzed and it is easier to computethe colors as they will appear under standardized viewing conditions.

Imagical 14 of a production node 104 may employ SpectraCube technology,or cameras using any filtering technology that can simulate StandardObserver response adequately. The preferred embodiment, imagical 14 hasone or more cameras, such as solid state area arrays, in which the lightcan be filtered through at least three wavebands, either simultaneouslyor sequentially, in conjunction with a unitary type colorimeter whichviews a determinable region of the image viewed by the camera. Thisprovides for the possibility of a spectral interpretation of each pixelas needed. A spectral interpretation is desirable in order to be able tocontrol color to a criterion of how it will appear under the finalviewing illuminant. For example, an on-press camera could be severalcameras/sensors, each equipped with a relatively narrow-band filter. Thecameras, used in conjunction with a unitary colorimeter, can samplecomposite spectral curves, as shown in FIG. 3B, in such a way thatinference of the total spectral reflectance function is possible. Theminimum number of camera channels depends on the number of colorants andthe complexity of their spectral absorption curves. The cameras mayinclude an infrared-sensitive camera to differentiate the blackcontribution to a spectral curve from contributions of non-neutralcolorants.

Preferably, imagical 14 is capable of multifocus or variable focusimaging so that larger areas of the image can be viewed at lowerresolution or vice versa. Views of small areas at high resolution enablesimulation of fine image structure by proofing devices capable ofsufficiently high resolution display. It is also preferred that imagical14 is equipped with anti-aliasing filters since much of the input to theimagical can be expected to be screened or pixilated. Also preferably,the viewing illuminant is controlled to simulate the viewing illuminantduring the measurement. This may be achieved using aspectrally-adaptive, electronically tunable filter to match theillumination spectrum of any desired light source.

Ease of use and accuracy requirements indicate that CMIs are calibratedor self calibrating in the preferred embodiment. The preferredembodiment of the unitary device SOM 13 approximates a dual-beam device,such as spectrophotometer of Van Aken et al. cited earlier. The spectrumof light reflected from the unknown sample is compared eithersimultaneously or successively with the light of the same sourcereflected from a known reflector. In this way it is possible to separatethe spectral contributions of colorants, substrates and illuminationsources and to estimate their true functional forms over multipleimpressions. This increases the CMI's accuracy. Even with suchmeasuring, however, regular recycling of instruments for factoryre-calibration or field recalibration using standardized reflectanceforms should be performed. Also, if video display 17 is a self-emissivemonitor, the above dual-beam functionality although not useful incomputing a difference spectrum, provides a means of evaluating possibledrift or need for re-calibration in the instrument.

System 100 operates in accordance with software operating at the nodes,which is preferably based on object oriented coding, a well knownprogramming technique. However, other programming techniques may also beused. The discussion below considers the different input and outputdevices, e.g., CMIs and rendering devices, as objects. Each objectrefers to software applications or routines in system 100 which providesinput to or accepts output from other applications (or devices).

Referring to FIG. 4A, a three-dimensional matrix model of the abstractclass device/profile is shown. For example, a device may be instantiatedas a linear input device or non-linear output device and deriveproperties through inheritance. An object created from a certain classis called an instance, and instances inherit the attributes of theirclass. Inheritance is a functionality of object-oriented coding andprovides for grouping objects having common characteristics into a classor subclass. The matrix is extended in a third dimension to include3-colorant-channel devices and more-than-3-colorant-channel devices.Display devices may have 4 or more colorant channels (for example, Red,Green, Cyan, Blue, indicated by RGCB Monitor 38) in order to enhancegamut or better simulate subtractive color reproduction processes. Inother circumstances, a display device may appear to fall into the sameclass as a CMYK printer 39, which it represents in asoft-proofing/client relationship. The appropriate models and transformsare those associated by inheritance with the subclass into which the newdevice falls within the general matrix.

A CMYK printer 39 exemplifies the (sub)classoutput/non-linear/more-than-three-channel. By inheritance,client-proofer relationships are shared by members of subclass output,since the ability to enter into client-proofer relationships can bepassed to all members of the subclass by inheritance. Note that thesubclass of input devices is distinguished by conservation of gamutamong its linear members. Likewise, specific instances of subclasslinear inherit an association with linear matrix models of colortransformation, whereas non-linear subclass members associate with colortransformation by polynomial evaluation, interpolation or other form ofnon-linear color mixture function. Procedures performing the colortransformations are incorporated in the data structures defining theobjects, and are discussed later in more detail. More-than-three-channeldevices require procedures for rendering with the extra colorants, whichis also discussed later in more detail.

Note that the class hierarchy depicted herein supports instantiation ofdevices of types “scnr” “mntr” and “prtr” as promulgated by the ICCProfile Specification. However, the class hierarchy disclosed herein isconsiderably more general, flexible and extensible. The properties offlexibility and extensibility are illustrated by the following practicalexample: a video display (“mntr” in prior art ICC Profile Spec) insystem 100 may occupy any of a number of cells within the classstructure depending on its physical properties (for example, the numberof colorant channels it has) and purpose (stand-alone design workstationor soft proofer representing a four-colorant device.)

The term “linear” herein, when applied to a device, means that linearmodels of color mixture can successfully be applied to the device. Itdoes not imply that a video display is intrinsically linear. Forexample, among input devices, linear defines that linear color mixturemodels can be used to convert from CIE TriStimulus Values to linearizeddevice signals and vice versa. Further note, that one application can bean output device with respect to another. For instance, applicationsoftware may convert RGB TriStimulus Values into CMYK colorants andoccupy the same cell as a CMYK printer 39.

The calibration of devices in system 100 is indicated by the classes ofthe devices 40, 41, 42 and 39 in FIG. 4A. Calibration herein includesthe process of obtaining color transformation data in uniform colorspace. Once devices at nodes in a configured network 11 of system 100are calibrated, the colorants produced by nodal rendering devices canthen be controlled, however such calibration remains subject torecalibration or verification processes, as described later. There arefour classes of devices 40, 41, 42 and 39 which require calibration:

1) imaging colorimeters or imagicals 14 (class input/linear/3-channel);

2) video displays 17 (generally in class output/linear/3-channel);

3) unitary, spectral colorimeters or SOM 13 (classinput/linear/more-than-3-channel); and

4) printers or presses (generally in classoutput/non-linear/more-than-3-channel).

Optionally, non-linear input devices may be used in system 100, but areless preferred. An example of a procedure for calibrating non-linearinput devices to a colorimetric standard is described in Appendix B ofthe American National Standard “Graphic technology—Color reflectiontarget for input scanner calibration” (ANSI IT8.7/2-1993).

In the first class of devices, the calibration of imagicals 14 involvespreparing compensation functions for the separable non-linearities ofthe device's transfer function, which are usually addressed in eachcolor channel individually. These compensation functions may be realizedin one-dimensional look-up-tables (LUT), one for each color channel. Thecompensation functions may be defined in a calibration step in whichmeasurement signals from imagical 14 are generated in response toobservation of step wedges of known densities. Next, specifying theconstant coefficients of a linear color mixture model expressed as a 3×3matrix transformation, hereinafter referred to as matrix M, whichconverts linearized device codes into CIE TriStimulus Values, such asXYZ, or related quantities. Last, the gamut of the input is scaled tofit within the color coordinate scheme in which images are represented.Because inputs to the system are printed copy (proofs and press sheets,)gamut scaling is often unnecessary except when the image representationis a space such as calibrated, monitor RGB which may not encompass allthe colors of the print. Occasions on which this would most likely be aproblem are those involving extra colorants used for “Hi Fi” effects,although limited ranges of conventional printing cyans and yellows areout-of-gamut for some monitors. Preferably, imagicals 14 areself-calibrating to assure the accuracy of their color measurements. Thecompensation function LUTs, matrix M, and possible gamut scaling dataare considered the calibration transforms for each imagical 14.

The color transformation in imagicals 14 into uniform color space basedon the above calibration is generally shown in FIG. 4B. Imagicals 14output measurement signals R, G and B, also referred to as device codes.The measurement signals are passed through compensation function LUTs 48(or interpolation tables) to provide linearized signals R_(lin),G_(lin), and B_(lin), in order to compensate for any non-linearities inthe relationship between light intensity sensed by imagical 14 and thedevice codes. Matrix M then operates on the linearized signals R_(lin),G_(lin) and B_(lin) to provide X, Y, and Z coordinates. Matrix M isshown consisting of the 3×3 coefficients (a₀₀-a₂₂) defining the linearcombinations of R_(lin), G_(lin) and B_(lin) needed to match X, Y and Z,the TriStimulus Values (TSVs) of the CIE Standard Observer. Although notshown in FIG. 4B, gamut scaling is performed after TSVs are converted toUniform Color Space coordinates such as those of CIELAB. Scaling of aninput gamut onto an output gamut in this case is entirely equivalent toprocesses detailed for rendering devices later in this description.

Calibration of video displays 17 in the second class of devices followsthe same steps for imagicals 14 described above, however since videodisplays 17 are output rendering devices, matrix M and compensationfunction LUTs are inverted.

Referring to FIG. 4C, device independent color coordinates XYZ are inputsignals to display 17. Rendering employs the inverse of the operationsused for imagical 14. The inverse of the calibration matrix M is calledA⁻¹ (to emphasize that we are considering numerically different matricesfor the two devices) and is used to convert the XYZ input signals tolinear device signals R′_(lin), G′_(lin) and B′_(lin). The linear devicesignals R′_(lin), G′_(lin) and B′_(lin) are postconditioned using theinverse of the compensation function LUTs which define the non-linearrelationship between applied signal and luminous output of display 17, afunction which is defined and adjusted in a separate, empirical step ofcalibration. The output from the LUTs are gamma corrected signalsR^(1/γ), G^(1/γ), and B^(1/γ) representing the input to display 17. Notethat there is no necessary relationship between the matrices A⁻¹ and Min FIGS. 4B and 4C. Further, since the LUTs of FIGS. 4B and 4C may beused with various types of transformations in system 100, they arepreferably represented by a separate data structure, within the softwarearchitecture, which may be combined like building blocks with otherstructures, such as 3×3 matrix, or multidimensional interpolation table,to form more complex data structures.

As stated earlier video displays 17 generally belong to the subclassoutput/linear/3-channel in FIG. 4A. However, there are two importantexceptions to this: a) when display 17 is used to represent amore-than-3-channel printer, the transforms used to drive the displayare non-linear—in other words, a proofing device manifests attributes ofits client; and b) when display 17 is used as an accomplice in thecreation of computer-generated art, the video display can be considereda linear input device, since new, digital, RGB data are created in themedium and color space of the display. Likewise, calibration for a RGCBMonitor (device 38 of class linear/output/>3-channel) is asimplification of procedures for calibrating the class ofnon-linear/output/>3-channel devices, which is described below.

In the third class of devices, the calibration of unitary, spectralcolorimeters or SOM 13 is set by the factory, as discussed earlier, andthus does not require the preparation of calibration data to providedevice independent color coordinates.

Referring to FIG. 5, the process of calibrating the fourth class ofdevices is shown. It should first be noted that for proofing devices inorder to represent one device on another (to proof) it is necessary tohave an accurate model of the color reproduction of both devices. It ispreferred that the proofing device has a larger gamut than the clientand that accurate knowledge of gamut boundaries be derivable from themodels of colorant mixture. Because proofing is an issue with outputdevices, it is necessary to develop rendering transformations thatinvert the models of colorant mixture and that mix colorants on theproofer in such a way as to match the colors produced by the client asclosely as possible. In other words, this should involve respecting thegamut limitations of the client device when rendering on the proofer. Insummary, the following four kinds of color transformations are developedin calibration of the fourth class of devices:

1) forward models that enable calculation of the color, in deviceindependent coordinates, of a mixture of colorants;

2) forward model inverses that enable calculation of the amounts ofcolorants needed to render a desired device independent colorcoordinate;

3) descriptions of gamuts in terms of boundaries specified in deviceindependent color coordinates; and

4) mappings of colors realizable on one device onto those realizable onanother in a common, device independent coordinate system (gamutconfiguration data).

The above four color transformations are discussed in more detail below.The following is in reference to a hard copy proofing device or proofer,but is applicable to other devices in the fourth class, including highand low volume presses.

Step 1 of FIG. 5 is the process of preparing linearization functions,which is shown in more detail in FIG. 6A. This process establishes alinear relationship between the digital codes sent to the proofer andthe output of the proofer, measured in units such as visual printdensity. Linearization improves the utilization of the available digitalresolution and is usually implemented by means of one-dimensional LookUp Tables (LUTs) that map linear digital codes from the nodal computeronto signals that drive the marking engine of the proofer to produce anoutput that is approximately linear. For example, step wedges printed inC, M, Y and K respectively should produce gradations in measurablevisual density that increase linearly as a function of the digital codescommanded by the host.

Usually, linearization involves printing a step wedge on the markingengine without the benefit of the LUT—if data are passed through theLUT, it applies an identity mapping. The color samples of the wedge areanalyzed by the CMI associated with the proofer and the measurements aresupplied to the nodal processor so that it can compute the transferfunction from command codes to print densities. The measured transferfunction is compared to the desired one and a function that compensatesfor the errors in the measured function is prepared—this is what isloaded into the LUT for use in normal image transmission. The LUTs arewritten to the local part of the VP as linearization functions.

Linearization is not a strict prerequisite for the remaining proceduresbecause a multidimensional color transformation could be designed toaccomplish what the one-dimensional LUTs are supposed to do. Thus,linearization of step 1 although preferred in system 100, may optionallybe incorporated into other color transformations in FIG. 5. However, itis generally advantageous to exclude as many sources of non-linearity aspossible from the transformation function which is specified by theprocedures outlined here.

Step 2 of FIG. 5 involves verifying or renewing the calibration of theCMI and rendering calibration forms, and is described in the flow chartof FIG. 6B. After initializing calibration procedures, if the CMIassociated with the rendering device is an imagical 14, it is calibratedto provide calibration transforms, as described above. Preferablycalibration of the CMI is performed automatically in response toinstructions from circuitry at the node.

After the CMI associated with the rendering device is calibrated, acalibration form is rendered on the proofer. For example, this form mayhave the following attributes: 1) a sampling of all combinations of fourlevels of all of the colorants, 2) inclusion of approximately neutralstep wedges, 3) several samples of flesh tones, 4) a number of redundantpatches—these are common inkings placed at different locations on theproof to provide information about spatial non-uniformities of theproofing process. It also is useful to include at least short stepwedges in each of the colorants and their overlaps, for example, cyan,magenta and yellow as well as blue (cyan+magenta,) green (cyan+yellow)and red (magenta+yellow.)

The calibration form described consists of about 300 samples in the caseof 4 colorants and can fit on an 8.5×11 inch (21.5×28 cm) sheet withpatch sizes of 1 cm on a side. However, generally, the number of patchesshould be three times the number of polynomial terms fitted to the data(discussed later in step 4, FIG. 8.) Patch sizes are scaleable forcompatibility with various CMIs. In addition to the tint samples, thetarget has markings similar to pin registration marks and demarcatedhandling zones to facilitate transfer of the target from the proofer tothe CMI if the CMI is not incorporated into the proofer. Theregistration marks indicate clearly where and how to insert the copy ina stand-alone CMI so that the instrument will find the patches wherethey should be and the handling zones will emphasize to the user thatthe image area should not be touched. Hard copy proofers may write anidentifying number and/or bar code for the device and for the specificproof (including date and time) on the proof.

Alternatively, a device in the fourth class, such as a running press,may be calibrated by analysis of live imagery (rather than a calibrationform) digitized by an imagical provided 1) that the images analyzedsample the gamut of the device adequately and 2) that the effects ofpage adjacency within a signature on color reproduction can be accountedfor (for example by reference to stored historical data at the node.) Asalways, the relevant data for calibration are the known colorantspecifications embodied on the printing plate and the colors resultingon the printed sheets.

After rendering the calibration form, the form is measured by a CMI andcalibration data is provided (Step 3 of FIG. 5). The processes of step 3are detailed in the flow chart of FIG. 7. As stated earlier, thepreferred CMI is a linear colorimeter, such as a imaging or unitarycolorimeter, which is capable of providing spectral analysis data whichsupports illuminant substitution. The CMI should produce severalreadings of color to the node; if one is an outlier compared to theothers because of a blemish on the copy or some malfunction, then thesoftware at the node excludes it from the averaging for that patch. Ifno two of the measurements agree, then the patch should be flagged so asto identify it as a problem in subsequent processing. More generally,the number of measurements of each patch is odd to permit voting by thesoftware in the interest of selecting trustworthy measurements.

If imaging of the calibration form is performed by an imagingcolorimeter or imagical 14, then the imaging colorimeter analyses theimages on the form, and uses its calibration transforms from step 2 tocalculate image colors and standard error of measurement in CIE UniformCoordinates, such as L*, a*, b*. Also the imaging colorimeter providesmany sample values of each patch color; regional checks fornonuniformity of color in the sampled area should be performed. However,if imaging of the calibration form is performed by an unitarycolorimeter or SOM 13, then the patch readings from the form areconverted to color measurements in CIE Uniform Coordinates.

Each patch measurement may include information about sensitivity,integration time, wavelengths sampled, illuminant substitutions, and thelike. The series of measurements from each calibration form areaccompanied by at least one record of the reference spectrum, although,obviously, reference spectral data will be collected and used on everyreading, at least for reflection measurements.

Regardless of the type of CMI, a list of colorant values (theIndependent Variables, IVs, to the fitting procedure) to correspondingcolor values (Dependent Variable) with standard deviation are assembled.On this list of measurements outliers are flagged. An estimate of sheetvariation from redundant sampling is produced.

In addition to multiple measurements within a given sheet, two othermeans for enhancing the reliability of the data are provided. First, thesoftware supports measurements of multiple sheets and second, anhistorical record of measurements from a particular proofer or press ispreferably maintained at a node. Historical data can be stored morecompactly and compared more readily to current data if the measurementsare converted from spectral form to colorimetric form. Spectral data ofthe measurement is stored in a database at the node in terms of colorand summary statistics.

Preferably, the database maintains a FIFO history of color and summarystatistics from the most recently measured forms. Because step 5involves least squares error minimization, flagging of outliers ispreferred to reduce the influence of one bad reading. A decision onwhether a current reading is legitimate is made by comparing CIE ΔE*values rather than the two spectra. The assembled list with flaggedoutliers and standard deviation of each patch measurement is written toa calibration (cal.) data file in the local part of the VP, for lateruse in building the forward model.

After step 3 is complete, processing continues to step 4 of FIG. 5,building a forward model based on the calibration of step 3. Step 4 isflow charted in FIG. 8. This model will represent color as a function ofcolorants of the proofer or press. It should first be noted that genericpolynomials provide a satisfactory form for models of color generationby colorant mixture on printing devices. However, this does not excludeany other mathematical or physical modeling procedure capable ofproducing transformation functions of sufficient colorimetric accuracy.Polynomials of relatively low order in each of the colorant variablesmay be fitted very nearly to within the device variation. In otherwords, the uncertainty of the model prediction is not much greater thanthe uncertainty of color rendered in response to a given set of digitalcodes. Low order implies that the colorant variables are not representedas powers greater than 2 or 3 and that the sum of the powers of theindependent variables in a single term of the polynomial is limited to4. Therefore, if the inks C, for cyan, M, for magenta, Y, for yellow,and K, for black are the independent variables, a valid term could beC²MK, but not C²M²K. Analytical derivatives are easily computed, whichmakes them advantageous for model inversion.

A polynomial forward model is fitted to a data set consisting ofindependent variables of colorant and dependent variables of deviceindependent color coordinates (stored in the calibration data file ofthe VP) by the method of least squares. A polynomial is a linearcombination of each of its terms which are called, mathematically, basisfunctions at step 82. Without loss of generality, discussion can besimplified by considering functions of two variables, C and M, in whicheach variable may be raised to a power of up to 2 and the sum of powersmay not exceed 4:R=a ₀₀ +a ₁₀ C+a ₂₀ C ² +a ₀₁ M+a ₁₁ CM+a ₂₁ C ² M+a ₀₂ M ² +a ₁₂ CM ²+a ₂₂ C ² M ²G=b ₀₀ +b ₁₀ C+b ₂₀ C ² +b ₀₁ M+b ₁₁ CM+b ₂₁ C ² M+b ₀₂ M ² +b ₁₂ CM ²+b ₂₂ C ² M ²B=C ₀₀ +c ₁₀ C+c ₂₀ C ² +c ₀₁ M+c ₁₁ CM+c ₂₁ C ² M+c ₀₂ M ² +c ₁₂ CM ²+c ₂₂ C ² M ²In the foregoing, color is the vector valued function |R G B| of thevariables C and M and the a's, b's and c's are constant coefficientswhich give the proportions of their corresponding terms to be mixed informing the linear combination. The purpose of the fitting procedure isto find the set of coefficients which results in the least squared errorwhen comparing the color measured at a given patch with the colorcalculated by substituting the patch's colorant values into the forwardmodel. The polynomials are untruncated within the constraints on thepowers. The DV may also be L*, a*, b* coordinates of CIELAB uniformcolor space.

In FIG. 8, the process of building a forward model begins by building adesign matrix for the problem (step 82). The design matrix has Mcolumns, one for each basis function, and N rows, one for each patchmeasurement. The number of rows should exceed the number of columns;practically, the ratio of N to M should be greater than 3 for goodresults. After reading the cal. data file, each cell in the matrix isfilled with the value of the basis function for the column atindependent variables (inks) given by the row, divided by the standarddeviation of the patch measurements, if available, else by 1 (step 83).Note that the patch measurements themselves do not enter the designmatrix. Then use the design matrix and vectors of patch measurements foreach of the three, color, dependent variable dimensions to write matrixequations that can be solved for the desired coefficient vectorspreferably by Singular Value Decomposition, SVD (step 84).

The numerical methods outlined in the preceding and the followingparagraphs are similar to those described by Press, et al. (NumericalRecipes, Cambridge University Press, Cambridge, UK, 1986.) Sections14.3, “General Linear Least Squares,” with SVD fitting, and 5.3,“Polynomials and Rational Functions”.

The model and its derivatives can be evaluated efficiently by a methodof recursive factorization. The method depends on use of polynomialterms as basis functions. However, it permits evaluation of the functionwith no more than 1 multiplication and 1 addition per polynomial term.Because the independent variables never need to be raised to a power,the demands for precision on the computational machinery are not asgreat. The principle can be seen most readily in one dimension; thefunction y=a₀+a₁x+a₂x² can be factored to yield a₀+x(a₁+a₂x) whichevaluates with 2 multiplies and 2 adds. Generalizing this to thetwo-dimensional function described above:a ₀₀ +a ₁₀ C+a ₂₀ C ² +M(a ₀₁ +a ₁₁ C+a ₂₁ C ²)+M ²(a ₀₂ +a ₁₂ C+a ₂₂ C²), ora ₀₀ +C(a ₁₀ +a ₂₀ C)+M[(a ₀₁ +C(a ₁₁ +a ₂₁ C))+(a ₀₂ +C(a ₁₂ +a ₂₂C))M].How to generalize to three or four dimensions is apparent.

At step 81, the patches that were flagged during measurement as outliersare excluded from the fitting. The fitting program estimates thevariation in the device based on deviations of color in redundantpatches and/or measurements of multiple copies. The average error of fitis calculated as the average ΔE* (CIE Color Difference Unit) over theset of measurements of patch colors compared to the colors predicted bythe fitted polynomial (step 83). This average should not exceed theestimated device variation by more than 1 ΔE* unit. When it does, orwhen the software detects individual patch discrepancies exceeding 4 or5 ΔE units, the software flags apparently outlying points (step 85). Itthen computes trial fittings with outlyers omitted to see if averageerror can be improved (step 86). It also is informed with a strategy forapplying various sets of basis functions in the interest of achieving anacceptable fit. However, the fitting procedure will reject a dataset atstep 86 rather than get too involved with it.

Principal Component Analysis, or an equivalent method, may be employedto reduce the number of polynomial terms (complexity of the model)consistent with a given average error criterion. This technique issimilar to those described in Johnson and Wichern, Applied MultivariateStatistical Analysis, 3rd ed., Englewood Cliffs, N.J.: Prentice Hall,1992, ch. 8.

Fitting concludes with writing a final polynomial model descriptor (adata structure and a file) consisting of a header and lists ofcoefficients. The header includes information such as the identity(ies)of proofs measured, relevant dates and times, the ingredient data files,the form of the polynomial (maximum powers of the independent variablesand maximum order of a term) goodness of fit statistics. The polynomialdescriptor will be needed later by the polynomial evaluator of thesoftware and is written into the shared portion of the Virtual Proof.

Although this discussion is directed to a 4-colorant printer or press,the polynomial forward model is, potentially, part of a soft proofingtransform which enables a video display to represent a client printer.It can be used to compute a colorant_(A) to colorant_(B) transformationfor what is effectively a CMYK, subtractive-color video display, in thatCMYK colorant values are processed through a transformation to producedevice-specific RGB for a monitor. This can generalize to use ofmore-than-four-colorant-printers and more-than-three-colorant displays.

After the polynomial model descriptors are computed, a forward modeltable (FMT) and prototype gamut descriptor data are prepared (step 5 ofFIG. 5). The processes of step 5 are detailed in the flow chart shown inFIG. 9A. As described above, the polynomial based on the modeldescriptors represents the forward model. The forward model enables usto predict the colors resulting when certain colorant mixtures are askedfor on the printer or press. System 100 includes a polynomial evaluatorto evaluate this polynomial either in hardware circuitry or software atthe node. Referring to FIG. 9B, a topology of operators is shown(multipliers 86 and adders 87) to implement the polynomial evaluator forthe two-colorant case, cyan (c) and magenta (m). Each operator receivestwo inputs. Inputs (operands) designated by two numerals are theconstant coefficients and inputs designated by C 89 or M 90 stand forthe independent variables, amounts of colorant. For instance, 21 88denotes the coefficient which corresponds to the second power of C timesM. In order to evaluate a function of three variables, 3 of the unitsshown in FIG. 9B are needed. Four variables require 27 such units for anuntruncated polynomial. In a hardware realization of the evaluator, itis preferable to retain the generality of the polynomial form and zeropoly terms (or their coefficients) that are missing due to truncation.If 27 units are too many for a cost-effective device, then thecalculation may be staged and the successive stages pipelined through asubset of the 27 units, albeit at some cost in control logic and speed.Given the opportunities for parallelism in a hardware implementation,the preferred embodiment benefits by a chip to transform ink values tocolors at video rates. Such a chip may be a component of nodal circuitryencompassing a graphics accelerator to drive a video display device forsoft proofing at the node. It accelerates the generation of colorseparation transformations because evaluation of the colorant to colormodel is a major factor in that computation. It also accelerates theevaluation of color separation transformations in situations where thedata of the inverse, color-to-colorant conversion can be fitted bypolynomials with a sufficiently small average error of fit. Data forfitting can be the addresses and entries of an interpolation table ofsufficient size, as discussed below.

The FMT stores the results of the polynomial evaluator in a datastructure and colorant quantization/addressing scheme shown in the CMYKhypercube of FIG. 9C, which contains the entire gamut of colorsreproducible by a CMYK printer. The hypercube may be subdivided into aFMT having 17 points (16 intervals) per colorant dimension forsufficient accuracy. However, the software architecture supports more orless, within the constraint that the numbers of grid points perdimension satisfy the equation 2^(n)+1, where n is integer. Depending onthe requirements of the rendering applications and equipment, a softwareswitch controls whether the tables are written in pixel interleaved(each address of a single table holds the values of all dependentvariables) or frame interleaved format. In the latter case, threeM-dimensional tables are prepared, where M is the number of colorants.Each “cell” of a table has an M-dimensional address and contains a colorcoordinate computed by evaluation of the forward model at step 97 ofFIG. 9A. At step 97 nested looping over all colorant addresses isperformed and computed colors from the forward model are stored at eachaddress. Thus, each model evaluation yields three color coordinates eachof which is deposited in a cell corresponding to the M colorants whichproduce it in the appropriate table. Colors of inkings in the midst of ahypercuboid are estimated by interpolation.

Referring now to FIG. 9D, each of the M channels of input to the FMT maybe provided with preconditioning LUT for each of the IndependentVariables and each output channel may be processed through a1-dimensional postconditioning LUT. Interpolation is preferablyperformed by linear interpolation in the nodal circuitry or software.

The data structure in FIG. 9D accommodates the preconditioning of jaddress variables by j 1-dimensional transformations which may beimplemented as look-up tables with or without interpolation 98. Thepreconditioning transformation may pass one or more of the j inputsIndependent Variables (IVs) through to the multidimensional transformunaltered (identity transformation) or may apply a functional mappingsuch as a logarithmic conversion. The multidimensional transform 94 hasj input variables and i outputs. The preferred implementation of thetransformation is by interpolation in a sparse table of function valuesor by evaluation, in hardware, of a set of polynomials fitted to thetabular values (where fitting can be done with sufficient accuracy.) InFIG. 9D, a 3-dimensional IV 93 is applied to the multidimensionaltransform 94. Multidimensional transform 94 consists of many smallercuboids 95 whose corner points are the sparsely sampled values of the Nat which values of one (or more) of the dimensions of the dependentvariable, DV, 96 are stored. The IV provides addresses and the DVcontents. The subcuboid 95 is shown at the origin of the addressingscheme. Interpolation is used to estimate values of the DV occurring atvalue of the N which are between corner points.

The data structure of FIG. 9D also accommodates the post-conditioning ofthe i output variables (DVs) of the transformation by i 1-dimensionaltransformations which may be implemented as look-up tables with orwithout interpolation 98. One of the transforming functions which may beincluded in the post-conditioning is the linearization functionoptionally produced by step 1 of FIG. 5.

The purposes of the 1-dimensional pre- and post-conditioning functionsinclude improving the extent to which the relationship between variablesinput to and output from the multidimensional transformation 94 isapproximated by whatever interpolation function is employed inevaluating the transformation. Therefore, the form of the functionsshould be known from preliminary studies of the device and the pre- andpost-conditioning transforms must be in place during calculations ofSteps 2 through 9 if they are to be used at all. For example, alinearization function which may be defined in step 1 should be used inrendering the calibration target in step 2.

A linear interpolation formula for two dimensions which generalizes tohigher dimensions with a simple, proportional scaling in the number ofoperations is described in Gallagher, “Finite element analysis:Fundamentals,” Englewood Cliffs, N.J., Prentice Hall, pp. 229-240, 1975.For example, given a cell in a two-dimensional array of sparsely sampledpoints as shown in FIG. 9E, the interpolated value of a function f(x,y)at a point (x,y) interior to the cell may be calculated as the weightedaverage of the endpoints in the direction of the longer of thecomponents x or y plus the weighted average of the endpoints in thedirection of the second longest component. In other words, order thedistances of the point with respect to the axes of the cell and then sumthe fractional distances along those ordered dimensions. Written as anequation:for y<x,f(x,y)=f(0,0)+x*(f(1,0)−f(0,0))+y*(f(1,1)−f(1,0)), andfor x>=y,f(x,y)=f(0,0)+y*(f(0,1)−f(0,0))+x*(f(1,1)−f(0,1)).

FIG. 9A also flowcharts the preparation of the prototype gamutdescriptor (GD) data. As colorant addresses are converted into colors topopulate the FMT (step 97), the colors are also converted intocylindrical coordinates of CIE hue angle, chroma and lightness. Hueangle and lightness are quantized to become addresses into a 2-D gamutdescriptor, preferably dimensioned at least 128×128 for adequateresolution (step 99). The Chroma component of the color is notquantized, but is stored in linked lists of Chroma values associatedwith each hue angle, lightness coordinate. The finished gamut descriptordata is prepared from the prototype GD data later at step 7 of FIG. 5,and consists only of surface chroma values for each coordinate.Therefore, the prototype is not a shared file and is written to thelocal portion of the VP, while the FMT is written to the shareableportion.

Next, at step 6 of FIG. 5, the FMT is inverted into a prototypetransformation table, also called Proto SEP table. Rendering image at arendering device requires this inversion step, i.e., finding thecolorant mixtures to realize a desired color which may be given indevice independent coordinates or in the color coordinate system of someother device (such as the CMYK monitor mentioned above.) Due to thecomplexity of the problem, it is generally not feasible to performforward model inversion in real time as imagery is rendered. Practicalapproaches rely on interpolation in a sparse matrix of inverse functionvalues. However, offering interactive control over key aspects of theseparation transformation (also called SEP) implies that at least someparts of the calculation occur nearly in real time. Because inversion ofthe forward model involves evaluating it, potentially numerous times,acceptable performance requires a very rapid means of evaluating theforward model. One method involves design of specialized hardware forpolynomial evaluation. When specialized hardware is unavailable,software performance can be significantly enhanced via the same strategyas used for speeding rendering transformations: interpolate in a sparsetable of forward model values, in particular in the FMT.

The proto SEP represents a color to colorant transformation with colorcoordinates as inputs and inkings as outputs. For illustration purposes,we will consider the case in which color coordinates are expressed inCIELAB units, L*, a* and b*. In order to build a sparse table of inversefunction values for the proto SEP, the following addressing scheme isdefined. If address variables, or inputs, are represented at 8-bitresolution, the span of the table in each dimension is 0-255 (2⁸)digital levels. Continuous perceptual dimensions are mapped onto thoselevels so as to satisfy two criteria: a) the entirety of the input andoutput gamuts of interest must be represented, and b) adjacent digitallevels are not perceptually distinguishable. Input and output gamutswill be detailed later; briefly, an input gamut is that of a device orapplication which supplies images (in this case for separation) and anoutput gamut is that of a device or application which receives theimage—in this case for rendering. With the above addressing scheme, aprototype separation table is computed. The following outlines thegeneral process for building the proto SEP table.

For each grid point, or address, of the table, the amounts of thecolorants needed to produce the input color are found. There are Ntables, one for each of N colorants, assuming frame interleavedformatting. First, a starting point of the correct inking is speculated.Calculate its color by the forward model and compare the calculatedcolor to the desired, address color. Save the color error, modify theinking, recalculate the color and see if the error is reduced. Theforward model is also employed to calculate the partial derivatives ofcolor with respect to ink. The resulting matrix, shown in the equationbelow, yields the direction of the movement in color occasioned by thechange in colorants. The partials of color error with respect to each ofthe colorants indicate whether movement is in the correct direction. Ifnegative, this indicates movement in the correct direction along thatink zero; then store each of the inkings in their respective tables.

$\begin{matrix}{\begin{matrix}{da}^{*} \\{db}^{*} \\{dL}^{*}\end{matrix} = \begin{matrix}{{{\delta\;{a^{*}/\delta}\; C},{\delta\;{a^{*}/\delta}\; M},{\delta\;{a^{*}/\delta}\; Y},{\delta\;{a^{*}/\delta}\; K}}} \\{{{\delta\;{b^{*}/\delta}\; C},{\delta\;{b^{*}/\delta}\; M},{\delta\;{b^{*}/\delta}\; Y},{\delta\;{b^{*}/\delta}\; K}}} \\{{{\delta\;{L^{*}/\delta}\; C},{\delta\;{L^{*}/\delta}\; M},{\delta\;{L^{*}/\delta}\; Y},{\delta\;{L^{*}/\delta}\; K}}}\end{matrix}} \\\;\end{matrix} \cdot \begin{matrix}{dC} \\{dM} \\{dY} \\{dK}\end{matrix}$

The foregoing paragraph gives a simplified explanation of the algorithmembodied in procedures which use Newton's Method for finding roots(“Newton-Raphson”) or one of many optimization methods for searching out“best” solutions in linear (e.g., the Simplex method of linearprogramming) or non-linear multidimensional spaces. The algorithm canalso be implemented with neural networks, fuzzy logic or with the aid ofcontent addressable memory. A neural network can be trained on theresults of repeated evaluations of a mathematical model. Further, neuralnetworks may also be utilized to perform any of the colortransformations of FIG. 5, e.g., building the forward model or renderingtransform table. Color transformation with neural networks is described,for example, in U.S. Pat. No. 5,200,816. Newton's method is applied tofind the root of an error function, i.e., the solution is at the pointat which the derivative of the error function goes to zero. Strictly,Newton's method is only applicable in situations where a solution isknown to exist. Therefore, although it is an efficient procedure, it issupplemented by other optimization methods, such as “SimulatedAnnealing.” (The model inversion technique described above is similar tothose in Press, et al. sections 9.6, 10.8 and 10.9, respectively,already cited.)

Optimization methods can produce a useful result when no exact solutionexists, as in the case when the desired color is unprintable by (or outof the gamut of) the device. The optimization is driven by an errorfunction which is minimized, typically by using downhill, or gradientsearch procedures. In the case of Newton's Method, one of the colorantvariables (in the four colorant case) must be fixed in order to find anexact solution; otherwise, there are infinitely many solutions. Usually,black is fixed. The algorithm for model inversion uses supporting searchprocedures in case the primary technique fails to converge to asolution.

The foregoing, simplified account overlooks the possibility that thegradient surface on the way to the solution has local minima in it whichthwart the search procedure. This risk is minimized by the presentinventors' technique of using the FMT to provide starting points whichare very close to the solution. The above process for building the protoSEP Table is applied in system 100 as shown in the flow chart of step 6in FIG. 10A. For each color entry in the FMT, find the closest coloraddress in the prototype separation table (step 111). Then use thecolorant address of the forward model table as the starting point in asearch for a more exact ink solution at that color address of the protoSEP (step 112). The search and the addressing of the interpolation tablecan be in the cylindrical coordinates of hue, chroma and lightness or inthe Cartesian coordinates L*, a* and b*. In the preferred embodiment,the forward model table has at most 17×17×17×17˜=84K grid points and theprototype separation table has at most 33×33×33˜=35K grid points many ofwhich will not be within the gamut of the printer and some of which maynot be physically realizable, i.e., within human perceptual gamut. (Thisis the case because the gamut of a set of colorants is not likely tohave a cuboidal shape suitable for representation in computer memorywhen it is converted from device coordinates to the coordinates of thecolor addressing scheme, as illustrated in FIG. 10B.) Therefore, thereis a favorable ratio of starting points to solutions.

An important advantage of keying off the FMT is that most searches areguaranteed to be for printable colors. For colors near the gamutsurface, Newton Raphson may fail because there is not an exact solutionsuch that an optimization procedure is required. The search routine mayuse either the interpolative approximation to the polynomial (the FMT)for speed in a software-bound system or the full-blown polynomial forgreater precision. In either case, the derivatives are easily calculableand the iterative searching procedures driven by gradients of colorerror (step 113).

The result of step 6 is a prototype color to colorant transformation(proto SEP) table in which virtually all printable addresses contain oneor more solutions. Because of the favorable ratio of starting points torequired solutions, many addresses will have solutions based ondiffering amounts of the neutral (black) colorant. The multiple blacksolutions are very useful in preparing to perform Gray ComponentReplacement and are stored in linked lists at the appropriate addressesof the proto SEP, which is stored in the local portion of the VP (step114). At this stage there are two ingredients of a final renderingtransformation in need of refinement: the prototype gamut descriptor andthe proto SEP.

An example of the results of step 6 is shown in FIG. 10B, in whichcoordinates which are plotted within a cuboidal structure suitable forrepresenting the perceptually uniform color space coordinates. Addresseshaving data are shown by black crosses. If the cube were subdivided intonumerous cuboids so as to create an interpolation table, as described inFIG. 9D, it is clear that many of the cuboids would not correspond tocolors that are realizable on the device. The coordinates of theperceptually uniform space are L*, u* and v* from the CIE's CIELUV colorspace in the example.

After step 6, the prototype GD data of step 5 is refined into finishedGD data at step 7 of FIG. 5. Step 7 processes are shown in the flowchart of FIG. 11. System 100 needs a favorable ratio of FMT entries toproto GD addresses; therefore, most of the addresses get one or moresolutions. Relatively few of these are surface Chroma points. Theobjective of step 7 is to use proto GD data as starting points for aniterative motion toward the gamut boundary using search and modelinversion techniques described previously. Gamut surfaces often manifestcusps, points and concave outward surfaces. Therefore, quantization ofthe descriptor must be at fairly high resolution (128×128 or more ispreferred) and initiation of searches for a surface point from othersurface points of different hue angle is often unfruitful. Because it isknown what colorant mix (i.e., little or no colorant) is appropriate atthe highlight white point of the gamut, it is best to begin refinementthere, using solutions at higher luminances as aids at lower luminances(step 121).

At least one of the colorants must be zero at the gamut surface.Therefore, the strategy is to start near the desired hue angle from wellwithin the gamut, move onto the desired hue angle and then drive outwarduntil one of the non-neutral colorants goes to zero (step 123). It isoften possible to zero the colorant that is expected to go to zero tohasten search procedures (such as Newton Raphson) which requireconstrained iterations. Starting points are usually available from alinked list stored in the proto GD; that failing, one may be had from aneighboring hue angle, lightness cell, or, that failing, by setting outfrom neutral in the desired hue direction (step 122). If polynomialevaluation hardware is available, it can accelerate on-the-flycalculation of surface points by forward model evaluation. Given a hueangle and lightness, the device coordinates bounding a small patch ofthe gamut surface are identified and colorant mixtures within the patchprocessed through the forward model until the maximum chroma at thehue/lightness coordinate is produced (step 124). In situations where itis necessary to fall back to methods involving model inversion, thehardware assist continues to be valuable because searching involvesnumerous evaluations of the polynomial and its derivatives, which arealso polynomials. Once a refined gamut descriptor has been completed, itis written to the shareable portion of the VP (step 125).

Referring to FIG. 12, a flow chart of the processes in Step 8 of FIG. 5is shown, filling in any holes which may exist in the PrototypeTransformation and computing functions that summarize the trade-off(Gray Component Replacement, GCR) between neutral and non-neutralcolorants at each color address. A “hole” is a color address which isprintable but for which no inking solution has yet been found. Theresulting proto SEP table is called a generalized color-to-coloranttable because it does not provide solutions for a specific amount ofblack. Step 8 has two goals. First, it seeks solutions for any colorsinterior to the gamut which have none (step 131) and writes NULL intoall unprintable addresses. The mapping of unprintable colors to withingamut occurs later in step 9 in response to gamut configuration data,which the user may select using the GUI screen of FIG. 21F, which isdescribed later. Second, it stores at each address the means to exchangeneutral for non-neutral colorants (Gray Component Replacement.) Thepreferred process is to store several solutions which can beinterpolated amongst using LaGrangian, Spline or generic polynomialinterpolating functions (step 132). The exact specification of blackutilization is incorporated later in step 9, which also may be selectedby the user via the GUI screen of FIG. 21E, which is also describedlater. The finished prototype color to colorant (proto-SEP)transformation table is written to the shareable portion of the VP (step133).

Step 9 of FIG. 5 includes the following processes:

-   -   1) Converting colorants of the proto-SEP transformation table        based on black utilization information specified in the black        color data;    -   2) Building Color to Color' Transform Table based on gamut        configuration data, e.g., gamut scaling, neutral definition or        gamut filter(s); and    -   3) Combining Color to Color' Transform Table with a        transformation table containing specific GCR solutions to        provide a rendering table.        The black color data and gamut configuration data may be set to        defaults or selected by the user as color preferences, as        described in more detail later.

Referring to FIG. 13, the processes of step 9 are flow charted. Althoughstep 9 is operative of four-colorant rendering devices, more-than-fourcolorant devices may also be provided with a rendering table as will beexplained in discussion of FIGS. 16A and 16B. First, the finished protoSEP table (generalized color-to-colorant) and the black utilization dataare read (step 140). The latter include Gray Component Replacement, %Under Color Removal (% UCR or UCR, a term referring to a limitation ontotal colorant application or Total Area Coverage, TAC) and possibleconstraints on the maximum amount of black or neutral colorant to beused; all three together are referred to herein as “black utilization.”

The prescriptions for black utilization come from one or more of thefollowing sources: a) local or system-wide defaults, b) broadcast ofcustom functions and limit values from a “boss node” configured innetwork 11 and c) values defined through a user interface which may beapplied locally or transmitted and shared. Note that modifications ofblack utilization do not have colorimetric effects. For example,compensating increases in neutral colorant by decreases in non-neutralcolorants in GCR does not change the color noticeably. Fields in the VPcontrol selection of the source of prescriptions for black utilizationunder particular rendering circumstances. A user may select the blackutilization in the Graphic User Interface of the model software, whichis described later in connection with FIG. 21E. The black utilizationdata, which includes % UCR, maximum black, and the GCR function or blacksolution, are stored in the shared part of the VP.

The first step in preparing a rendering transformation, then, is toconvert the data on GCR stored at each printable address into aparticular black solution by referring to the curve giving % GCR as afunction of density while observing the maximum black constraint (step149). Thus, the entries in the proto-SEP transformation table areconverted based on a specific GCR solution within a maximum black limit.The converted proto-SEP table is stored in the local part of the VP. Thesecond step is to find the maximum neutral density at which the totalarea coverage limitation is satisfied, i.e., % UCR (step 150). This isdone by moving up the neutral, or OE Lightness, axis through thequantization scheme from the minimum printable Lightness, examiningcolorant solutions, until one is found that is within the limit. Theminimum Lightness so identified is stored for use in the gamut scalingprocess to be discussed presently. Although it is conceivable to storemin Lightness values as a function of color, rather than simply as afunction of Lightness, for purposes of gamut scaling, this is usually anunwarranted complexity.

In order to convert the GCR-specific result of the foregoing methodologyinto a rendering transformation, it must be “married” with a“conditioning” transformation. The latter is a color-to-color'conversion expressible as an interpolation table of the format presentedearlier. (Note that any transformation that is evaluated byinterpolation and that can be fitted with acceptable color accuracy by apolynomial may be performed by suitable hardware for polynomialevaluation.) It serves the multiple purposes of scaling unprintablecolors of the addressing scheme onto printable values, “aliasing” thecolor definition of neutral to accommodate user requirements (FIG. 21E,270) and effecting conversions among color addressing variables.

An example of the latter is the conversion from Cartesian CIELABaddressing to “Calibrated RGB” addressing which is useful if the imagedata that will be processed through a rendering transformation arerepresented as RGB. It will be described later that ConditioningTransformations (CTs) play an important role in verification and devicecontrol. A CT is often the result of the “concatenation” of severalindividual, conditioning transformations serving the purposes justidentified. The applications of transform concatenation include: 1) themethod whereby a separation table with many NULL entries is made usefulfor rendering by concatenation with a conditioning transform whichperforms gamut scaling with considerable savings in transform-generationtime and 2) feedback control of the color transformation process as willbe detailed later.

In order to minimize cumulative interpolation errors, intermediate colorto color' transforms may be stored and/or evaluated at high precision.Wherever possible, exact conversions of table entries should beutilized. For example, once all the mappings of CIELAB color to CIELABcolor' have been compiled, the conditioning table entries for a colorcoordinate conversion from CIELAB to “calibrated RGB,” for example, maybe computed using analytical expressions.

Note that system 100 does not preclude importation (as described indiscussion of User Interface) of standardly formatted colortransformations (“profiles”) prepared by applications other than theVirtual Proofing application. Accordingly, the rendering transformationmay incorporate user preferences indicated through atransformation-editing tool (called a “profile editor” by someApplication Software Packages.)

A key purpose of the conditioning transformation is gamut scaling,described below. It is important to elaborate this point: If the targetdevice for the rendering transformation is a proofer, then the outputgamut to which colors are scaled may be that of its client. Therelevant, conditioning data for all devices on the network resides inthe Virtual Proof. If the client's gamut fits entirely within theproofer's, then substitution of client for proofer gamuts is used torender a representation of how imagery will appear on the client. Thedisplay and negotiation of compromises to be made when the client'sgamut does not fit entirely within the proofer's is discussed inconjunction with FIG. 21F, below. Note that, in addition toaccommodating a client's gamut, successful proofing may require othermappings. For instance, tone scale remappings to compensate fordifferences in overall luminance levels between video displays andreflection media may be performed. Likewise, “chromatic adaptation”transformations to compensate for changes in illumination, viewingconditions, etc. may also be implemented by means of data structures ortables of what are called “conditioning” or color-to-color' transforms(output color transforms) herein.

Elaboration of the concept of color “aliasing,” presented above is alsoappropriate: Neutrals are conventionally defined in terms of colorant inindustry; elements of the default neutral scale which appears in FIG.21E (step 270) generally do not have common chromaticity coordinates upand down the Lightness axis. In order to map colorimetrically neutraladdresses onto the desired mixtures of colorants, it is necessary toconvert the colorimetric addresses to the colors of the “colorantneutrals”—a process herein referred to as “aliasing” because one coloraddress masquerades as another. The term has a different usage, familiarto signal processing, when used herein in connection with imagingcolorimetry.

Specifically, the processes involved in making a color-to-color'transform (numbered 1 to 4 in FIG. 13) are as follows: (Recall that thegoal is to be sure that only printable address colors are applied to theGCR-specific SEP table in the course of making a Rendering Table.Normally, color_(out)=color_(in) at the outset. More detailed discussionof input gamuts, output gamuts and gamut operators follows. The sequenceof the processes is important.)

1) Negotiate gamuts: In preparing a rendering transform to enable aproofing device to represent a printing press the color addressing ofthe color-to-color' table may be defined by the input gamut—in terms ofthe color image data. The range of colors represented in the table islimited by the extreme image values in the three dimensions of UniformColor Coordinates. Because the quantization scheme is regular (cuboidal)there will be color addresses which do not occur in the image data(recall FIG. 10B.) These may be ignored, or mapped approximately ontothe surface of the image's gamut. In place of the image's gamut, ageneral representation of the gamut of the original medium of the imagemay be used.

This method is preferred, for instance, to using the gamut of the pressas the input gamut because so doing would require conversion of bulkyimage data into coordinates that are all within the press's gamut. Anobjective of system 100 is to provide means of interpreting image datain various ways without creating a multiplicity of versions of thatdata. An exception would occur if image data specifically from a presssheet were required to evaluate image structure due to screening, etc.as part of the proofing process.

The output gamut may be the lesser of the client's (printing press) ANDproofer's gamuts, where “AND” connotes the Boolean operator whose outputis the “least common gamut.” This may be derived using gamut filters, asdiscussed later. In this negotiation, the proofer's gamut may beconstrained to be that available within the Total Area Coverage (% UCR)limitation applicable to the press at the user's discretion. Otherinteractive tools may be used to control the definition of the outputgamut subject to the ultimate restriction of what can be rendered on theproofing device within the default or selected % UCR constraint, ifapplicable. (Of course a video display proofer's gamut is notintrinsically subject to a UCR constraint.)

2) Perform gamut scaling: Using gamut operators to be discussed later,map color values to color' and store the color' values at the coloraddresses in the table.

3) Perform neutral aliasing: In each lightness plane, the color of theconventionally neutral inking (FIG. 21E, 270) is offset from the neutralcolor coordinate a*=b*=0 by an amount a*,b*. In order to map imageneutrals to this offset color, the color' values in the conditioningtable should be shifted in such a way that an address of 0,0 maps to acolor' value of a*,b*. The function which performs the shift may bedesigned so that the amount of the shift decreases with distance fromneutral.4) Transform Color Coordinates (if necessary): The reason for this and amethod of implementation were suggested previously, namely, it ispreferred to perform gamut operations in Uniform Color Coordinates, butthe image data may be represented in a color notation such as“calibrated RGB” so that a rendering table must be addressable by RGBcoordinates. Because exact mathematical equations generally govern therelationship between Uniform CIE color and calibrated RGB, each color'value in the conditioning table is converted to RGB by means of theequations.

The Color-Color' Transform (XForm) is produced by the above steps andthen concatenated with the GCR-specific SEP table. The result is arendering table for transforming the color of input color image datainto colorant data for the rendering device, which is stored in thelocal part of the VP.

The following discussion is related to providing gamut mapping data ofthe gamut configuration data. Color addresses that are not printablemust be mapped onto printable ones. In the present application, theoutput gamut is that of the proofer of interest or the printer itrepresents. However, the input gamut is not fixed by the architecture orsoftware design because it may vary and have a profound effect on therendering of out-of-gamut and nearly-out-of-gamut colors. This is thecase because the outermost, or limiting, colors vary greatly amongpossible input gamuts. Input gamuts that warrant distinctive processinginclude:

1) Other proofing devices include both hardcopy and video displaydevices (VDDs). Hardcopy devices are likely to have gamuts that arefairly similar, requiring only minor relative adjustments.Self-luminescent, additive-color devices such as video displays havevery differently shaped gamuts from reflection devices so that the bestmapping from input to output as a function of color will be unlike thatfor a reflection device. Computer-generated images originate inapplications which are likely to exploit the gamut of a video device.Many retouching and page assembly applications use the RGB coordinatesystem of the monitor to store and manipulate images because itfacilitates display on the VDD and interactivity. In this case, theinput gamut is often that of the VDD, even if the image was originallyscanned in from film.

2) Printing presses will usually have smaller gamuts than the proofingdevices that represent them, restricting what is to be used of theavailable proofing gamut. If printed images are captured by an imagingcolorimeter as part of calibration or verification so as to constitutethe image data, the input gamut may be better approximated by therendering gamut of the printer than the receptive gamut of thecolorimeter.

3) Electronic or digital cameras will usually have much greater gamutsthan any output device, necessitating a very significant restriction onwhat portions of the input gamut can be printed. Note, however, that themaximum gamut of a linear camera encompasses many colors that are notcommonly found in natural photographic scenes. Accordingly, it may bepreferable to design mapping functions for this class of device that arebased on the scenery as well as ones based on device capabilities.

4) In conventional photography, the scene is first captured on filmbefore being captured digitally. The relevant input gamut is therendering gamut of film.

5) Regardless of the input medium and its gamut, there may beimage-specific imperatives. For instance, a very “high-key” image,consisting almost entirely of highlights, lace, pastels, etc. may notmake extensive use of the available gamut of a color reversal film. Thebest gamut mapping for this particular image is not the generic one forfilm.

The gamut mapping data is provided by a gamut operator which is afunction which maps an input color to an output color. The process ofconstructing the gamut operator is shown in FIG. 14. It is a commonpractice to “clip” the input gamut to the output. In other words, allcolors outside the output gamut are mapped onto its surface. This may bedone in such a way as to preserve hue, saturation (Chroma) or Lightnessor a weighted combination of the three. System 100 can work with suchoperators and supports access to them through the “Rendering Intents”function in the GUI, as shown latter in FIG. 21F. However,invertibility, reciprocality and smoothness are preferred properties ofgamut operators, especially when processing and transforming image data.

Invertibility is an important property of the function because itinsures that no information is lost except that due to quantizationerror. Reciprocality means that a mapping of input color to output colormay involve either a compression or reciprocal expansion of the gamut,depending on which gamut is larger. Smoothness, or continuity of thefirst derivative, reduces the risk of noticeable transitions (“jaggies”)in images which are due to gamut scaling. A simple exemplar of aninvertible, reciprocal operator is illustrated in FIG. 14. It ispresented to demonstrate and clarify key concepts and then an operatorwhich is also smooth is explained. A mapping of Psychometric Lightness,L*, is shown, however, the same operator is applicable to CIE Chroma,C*, as well. It is assumed that hue is to be preserved in the mapping; aseparate tool is supported within the GUI to gamut operations latershown in FIG. 21F, for situations in which users want to modify hues.FIG. 14 depicts the two cases of reciprocality, one in which the dynamicrange of the input device must be compressed in order to fit the outputgamut and the other in which input lightnesses can be expanded to fillthe larger dynamic range of the output device. The operator introducedbelow is able to handle both cases without explicit user intervention,although operator override is also supported through the GUI of FIG.21F.

Define L_(pivot) as the greater (“lighter”) of the minimum input andoutput L* values.L _(pivot)=max(L _(min_in) ,L _(min_out))where the minimum input lightness may, for example, be the L* value ofthe darkest color that can be reproduced on a positive reversal film andthe minimum output lightness the L* value of the darkest color printableon a reflection medium. L_(pivot) is denoted in FIG. 14A by a plaindashed line.

For lightnesses higher than L_(clip), the gamut operator maps input tooutput lightnesses identically as indicated by the equation in the openspace below the maximum L* of 100. A “cushion” is put between L_(pivot)and L_(clip) in order to insure that the mapping is invertible:L _(clip) =L _(pivot)+(L* _(max) −L _(pivot))*cushion.0.1 is a reasonable value for cushion, chosen so as to reduce the riskof losing information due to quantization error to an acceptable level.In the limit in which cushion=1, the entire range of input L* values isscaled uniformly, or linearly, onto the range of output L* values.

In either case 1 or 2, all lightnesses between L_(clip) and L_(min_in)are scaled onto the range of lightnesses between L_(clip) andL_(min_out), whether the scaling represents a compression or anexpansion. A piecewise-linear scaling function is illustrated below forsimplicity. Note that all L values refer to CIE Psychometric Lightnessin these equations whether they appear with a * or not.

If ( L*_(in) > L_(clip) ) then L*_(out) = L*_(in) Else L*_(out) =L_(clip) − [(L_(clip) − L*_(in))/(L_(clip)−L_(min) _(—) _(in)) *(L_(clip)−L_(min) _(—) _(out))] .

The concept can be extended by adding the property of smoothnessdescribed earlier. The operator is based on a continuouslydifferentiable function such as sine on the interval 0 to π/2, whichgenerally resembles the piecewise linear function described above inshape but has no slope discontinuity. A table (Table I) of values of thefunction is given below; the first column is a series of angles inradians, X, from 0 to π/2 (90), the second, the sine of the angle, Y,and the third the fraction Y/X. If we set Y/X=(1−cushion), we cancontrol the “hardness” or abruptness of the gamut-mapping implemented bythe operator stated below the table in for the case of cushion˜=0.1. Forspeed, the various evaluations implied may be implemented byinterpolation in look-up tables. The operator described does not enablea purely proportional scaling (cushion=1.) The latter is not generallydesirable but is available to users through the gamut options of the GUIin FIG. 21F.

TABLE I Angle, x (rad) sin(x) sin(x)/x 0.0000 0.0000 * 0.0873 0.08720.9987 0.1745 0.1737 0.9949 0.2618 0.2588 0.9886 0.3491 0.3420 0.97980.4363 0.4226 0.9686 0.5236 0.5000 0.9549 0.6109 0.5736 0.9390 0.69810.6428 0.9207 0.7854 0.7071 0.9003 <------ “cushion” 0.10 0.8727 0.76600.8778 0.9599 0.8191 0.8533 1.0472 0.8660 0.8270 1.1344 0.9063 0.7989 If( L*_(min) _(—) _(in) < L*_(min) _(—) _(out)) .707 * ( (100 − L_(out)) /(100 − L_(min) _(—) _(out)) ) = sin [.785 * ( (100 − L_(in)) / (100 −L_(min) _(—) _(in)) ) ] Else .785 * ( (100 − L_(out)) / (100 − L_(min)_(—) _(out)) ) = arcsin [.707 * ( (100 − L_(in)) / (100 − L_(min) _(—)_(in)) ) ]

The operators presented above rely on gamut descriptors to find thelimiting colors of the input gamut that must be mapped into the outputgamut. Once corresponding surface points in the two gamuts have beenidentified, the scaling function is used to prepare the conditioningtransformation.

In summary, input gamuts can be very different from output gamuts. Theyoften have a larger range of luminances (or dynamic range) such that itis preferable to perform a scaling in Lightness before working on Chromaor saturation. Secondly, scaling of Chroma is performed along lines ofconstant hue. Compensation for imperfections in the CIE models (embodiedin CIELAB and CIELUV) of hue constancy or for elective changes in hueare handled separately. An example of elective changes is the need tomap highly saturated yellows from film input to highly saturated printyellows by way of a change of hue angle. Modifications of hue can beaccomplished through the conditioning transformation by color aliasing.

Referring now to FIGS. 15A and 15B, the shareable and local componentsdescribed above in the VP are shown. In the interest of compactmessages, not all shareable data need be transmitted in each transactioninvolving Virtual Proof. To the application software which administersthe Graphical User Interface Software, both operating at a node, the VPis a set of data structures based around the classes Node andDevice/Transform. The data structures map onto a general file structurewhich is not bound to a particular operating system, computer platformor processor. Object-oriented conventions of data-hiding are employed inthe software to insure the integrity of transformations which aremanipulated at a node. Accordingly, the VP files which store the dataand transformations have local and shared components, as stated earlier;shared components consist of data which are read-only to all nodesexcept the one assigned responsibility for the data. Duringinitialization in preparation for virtual proofing described inconnection with FIG. 18, participating nodes insure that appropriatedata are written to the relevant nodal fields within the shared filesystem.

The VP enables revision of color aspects of page/image data up to andeven during rendering. An important aspect of revisability is thecustomization of data for rendering on particular devices. An equallyimportant property of revisability is that page/image data need not bealtered directly; rather they are “interpreted” in various ways throughthe medium of the VP. This eliminates the need to maintain multiple,large versions of page/image data at a particular node or to move one ormore versions around the network repeatedly as a result ofre-interpretation. Therefore, although the VP allows for imagespecificity and linking, preferably it is not bound into page/imagefiles. The structure of the VP is similar to that of the Tagged ImageFile Format (an example is described in “TIFF™ Revision 6.0”, Jun. 3,1992, Aldus Corp., Seattle Wash., pp. 13-16).

An example of the tagged or linked list file format for the shared partsof the VP is shown in FIG. 15C. The tags are referred to as Field IDs(bottom right of FIG. 15C.) It is possible for a given device to bepresent in a VP file more than once, specialized to represent images fordifferent input gamuts or black utilization, etc.

System 100 may incorporate rendering devices at nodes having more thanfour colorants. The processes for performing color to colorantconversions for more than four colorants is shown in FIGS. 16A and 16B,which are connected at circled letters A and B. In FIG. 16A, afterstarting, if the extra colorant is neutral, then proceeding continues tothe start of FIG. 16B. Otherwise the following steps for addingadditional non-neutral colorants (e.g., Red, Green and Blue) areperformed:

Step 1) Proceed as for 4 inks through to the stage of gamut scaling. Usethe Black Utilization tool which enables % GCR to depend on Chroma, topush towards maximum Black (2-colors+Black, with the complementary colorpushed toward zero) solutions. Save this as an intermediate table. Thisintermediate is the equivalent of a GCR-specific SEP table.

Step 2) Build a model for converting C, M Blue and K (black or N) tocolor and omitting the colorant complementary to the “auxiliary,” inthis case, Yellow. Make a Forward Model Table and use the model toextend the original gamut descriptor prepared in (a). Do likewise for C,Y, Green and K and M, Y, Red and K. Note that the general rule is to addadditional colorants one at a time, grouping each with the colorantswhich flank it in hue angle. Make FMTs for each new model for eachauxiliary colorant and re-refine the Gamut Descriptor. Note, however,that the multiple models are used to refine only one GD.

Step 3) Modify the proto-rendering table (only one of these ismaintained): Within the C,M,Blue,K gamut, there are not multiplesolutions with different amounts of black at a given color; however,there are many solutions trading off CM vs. Blue. Store linked lists ofthese solutions at relevant color addresses in the intermediate table.Do likewise for CYGreenK and MYRedK.

Step 4) Treat the intermediate table as a “prototype table” in the4-colorant case. Perfect it by making sure that all printable addressesin the new color regions of the table have at least one solution (“fillthe holes.”)

Step 5) Once the intermediate table has been reprocessed for allnon-neutral auxiliary colorants, convert to a rendering table byperforming the analog of GCR in the three new regions of the table.Check total area coverage, re-scale gamut and complete as for four inks(Step 9, FIG. 5.)

The foregoing procedure does not estimate the full gamut available; forexample, the gamut at the hue angle of cyan is increased by theavailability of blue and green. In other words, the BCGN gamut (where Nstands for Neutral, or black) is not considered in the foregoing.Overprints of Blue and Green are likely to be substantially dark,compared to cyan. Therefore, the additional colors available in thisgamut are not, in general, very numerous and need not be computed.However, in those cases in which the lost colors are important, theprocedure outlined above is extended to include auxiliary gamutscentered on the standard subtractive primaries (C, M and Y) rather thanon the additional colorants (R, G and B.) The result is overlappingauxiliary gamuts. By default, the decision regarding which of theoverlapping auxiliary gamuts to search for a solution chooses the gamutcentered on the ink of closest hue angle. There is nothing in theprocedure which prevents its extension to even more colorants, which maybe added in a recursive fashion. However, practical applicationsinvolving the overprinting of more than 7 (or 8, in the case of an extraneutral) colorants are very unlikely.

After the above steps are complete, if more colorants need to be added,processing branches to the circle A at the start of FIG. 16A, otherwisethe process for adding additional colorants is complete. In the case ofaddition of auxiliary colorants which does not involve overprinting morethan 3 or 4 colorants at a time (as in the case of multiple, “custom”colorants that might be used in package printing) the colorants aretreated as separate sets according the procedures outlined previously.

If the process branched to FIG. 16B (to circle B), then the followingsteps for adding an approximately neutral colorant, such as gray, areperformed: If additional non-neutral colorants are also to be added, addthem according the procedure outlined in FIG. 16A above.

Step a) Prepare a colorant to color transformation for the 5-colorantset CMYKGray. Evaluate this model either with polynomial hardware orwith a 9×9×9×9×9 interpolation table having about 60,000five-dimensional cells. The simple linear interpolator is preferred andis particularly appropriate to this situation because the complexity ofcalculations scales linearly with the dimensionality of theinterpolation space. As usual, make a Gamut Descriptor in tandem withbuilding FMT.

Step b) Invert the model as in the 4-colorant case, fixing black andgray; build up linked lists of alternative solutions for a given color.

Step c) Proceed as in the 4-colorant case. When inverting the model, useonly CMY and Gray wherever possible (i.e., fix black at zero,) addingblack only as necessary to achieve density. There are two stages of GCR.In the first, black is held to a minimum and gray is exchanged with C, Mand Y. In the second, black may be exchanged, optionally, with gray andsmall amounts of the other colorants, as needed to keep the colorconstant. In the second stage, an error minimization routine is needed;Newton-Raphson is not appropriate.

Step d) UCR, preparation of a conditioning transformation, and so on asin the 4-colorant case, follow the second stage of GCR. Complete therendering table, except for addition of auxiliary, non-neutralcolorants.

After the above steps a-d, if additional non-neutral colorants are alsoto be added, processing branches to circled A in FIG. 16A, otherwise theprocess for adding additional colorants ends.

Referring to FIG. 17, the process for building a gamut filter is shown.The finished prototype color to colorant table, filled with NULL cellsand specific GCR solutions, is one manifestation of a device's gamut. Itcan be converted into a very useful filter in which each cell gets anindicator bit, 0 if the cell is NULL and 1 otherwise. The filters of twoor more devices can be used to enhance visualizations.

Many graphics cards for driving video displays provide an alpha channelor overlay plane enabling the visualization of translucent graphicalinformation on top of a displayed image. The results of performingBoolean operations on combinations of gamut filters for multiple devicesmay be converted into color-coded or pseudocolor overlay information toreveal things such as which image pixels are in gamut for one device butnot another. With this tool, the intersection of the gamuts (the “LeastCommon Gamut”) of five presses can readily be compared with each press'sgamut in turn on common imagery, in support of a decision about what touse as a common criterion for color reproduction across the network.

Semi-transparent overlays are generally not possible for hard-copydevices without extensive image processing. In the case of a printer, amonochrome version of the image may be prepared and printed, overlaidwith colored speckles indicating regions of overlap of multiple gamuts.The “overlay” is actually constituted by redefining subsets of thepixels in the image which belong to a certain gamut category as aparticular speckle color. The foregoing method involves making amodified copy of the color image data with reference to the gamutfilter.

An alternative, and preferred, method provided by the invention forreadily identifying gamut overlaps is to filter the color-to-color'transform or actual rendering table. This may be done by leaving thecontents of “in” addresses as they were while modifying the color orcolorant contents of one or more varieties of “out” addresses to containwhite or black or other identifying color. Different identifying colorsmay code different regions of intersection, overlap or disjointedness ofthe gamuts of several devices. When one or more channels of the(original) color image data are rendered through the “filtered renderingtable,” colors in the image which are out of gamut are mapped to one ofthe identifying colors and the resulting print reveals gamut limitationsof various devices with respect to the imagery itself. An additionaladvantage of this method is that it is effective even when some of thecolors under consideration are out of gamut for local proofing devices.

Another method of visualization available with the filters is to look atslices through Boolean combinations of the filters for two or moredevices. Finished proto SEP tables are not generally useful other thanin the preparation of rendering tables for a particular device;therefore, they are kept local. The filters are written to the sharedportion of the VP. More specifically, in FIG. 17 a flowchart of theprocess of making a gamut filter in either a compressed form, in which asingle bit is 0 or 1 to denote NULL or “in-gamut,” or in a form in whicheach color's status is coded with a byte equal to 0 or 1 is illustrated.In the compressed case, a computer word may be used to represent a rowor line of the prototype color-to-colorant table with each bit packed inthe word representing one color address. After reading the proto-table(step 1,) the steps of making the compressed filter include 2c) loopingover the lines of the table, 3c) setting or resetting the bits of a worddepending on printability and 4) finally writing the filter to shareableVP. The steps for making a byte table are completely analogous, exceptthat a byte is devoted to each color address.

The two basic processes of calibration and verification (monitoring)rely on instrumentation and interact to produce the rendering transformwhich embodies the VP and can interpret color image data.

Referring now to FIGS. 18A-B, a flow chart of the program operating atnodes 102 and 104 for virtual proofing in system 100 is shown. Thesefigures are connected at circled letters A-D. At the top of FIG. 18A,one or more users invoke the application software 1801; a single usercan run the program in order to revise the Virtual Proof as it relatesto the local node, or to other nodes which are accessible. Often,multiple users run the program simultaneously to negotiate the VP.Network readiness is established by making sure that the relevant,participating nodes all share current data 1802. CMI's are put through(preferably automatic) calibration checks 1803. Next, verification ofdevice calibration is attempted by rendering and analyzing color 1804.The details of this step depend on the nature of the device and of thecolor measurement instrument; if the device is a press, particularly onewith fixed information on the plate, the verification is most likely toinvolve “live” imagery rather than a predefined cal/verification form ortarget such as one consisting of tint blocks. The color error data 1805produced by verification are supplied to the press controls, ifappropriate 1806, as well as to the program to support decisions aboutwhether color variations can be compensated by modification of existingrendering transforms of the VP or whether recalibration of the device isrequired 1807.

If re-calibration is called for, the program branches to C, atop FIG.18B, where systematic calibration after FIG. 5 is undertaken 1808. Else,the program branches to B, atop FIG. 18B, to revise the color-to-color'transform 1809 based on the processing of the color error data, which isdetailed later in FIGS. 19 and 20. Next, the need for User-preferencerevisions is assessed at D. If YES, then gather User preference data1810 and re-specify rendering transforms 1811 as in Step 9, FIG. 5. IfNO, then revise VP for a new interpretation of image data and render1812. If results are satisfactory, conclude. Else, either recalibrate atA or revise preferences at D, depending on the nature of thediagnostics.

Verification is a feature of system 100 used in virtual proofing,described above, and color quality control of production renderingdevices. The reason for verification is that the use of system 100 forremote proofing and distributed control of color must engenderconfidence in users that a proof produced at one location lookssubstantially the same as one produced in another location, providedthat the colors attainable by the devices are not very different. Oncerendering devices are calibrated and such calibration is verified toeach user, virtual proofing can be performed by the users at therendering devices. In production control, such verification provides theuser reports as to status of the color quality.

During verification of production rendering devices, on-press analysisof printed image area may be used in control of the production processand for accumulation of historical data on color reproductionperformance. Historical data may be used in a statistical profile of themanufacturing run which serves as a means of verifying the devicecalibration. It is also used to inform and update the virtual proof,enabling better representation of the production equipment by a proofingdevice. With a sufficient accumulation of historical information, it iseven possible to model and predict the effects of neighboring pages in asignature on the color in the page of interest to an advertiser.

Once a device has been calibrated, the color transformations thus can beone of the mechanisms of control. In a control loop, colors produced bythe device are compared to desired values and mechanisms affectingcolorant application are modulated to reduce the discrepancy betweenmeasured and desired values. Control implies continuous feedback anditerative adjustment over many printing impressions, whereas proofingdevices are usually one off. Nevertheless, proofing devices vary withtime and periodic recalibration is a means of control.

One feature of system 100 is to provide the User with information aboutthe color accuracy of the proof. It has been noted that the invention iscompatible with two kinds of instrumentation, unitary colorimeters (SOMs13) capable of measuring homogeneous samples and imaging colorimeters(imagicals 14) capable of sensing many pixels of complex image datasimultaneously. In the following discussion, differences in verificationprocedures for the two kinds of instrument are considered.

Calibration is best carried out with a specialized form which is knownto explore the entire gamut of the device. The rendered form can bemeasured by either type of instrument. In verification, the requirementsof sampling the entire gamut are not as stringent; the interest is oftenin knowing how well the reproduction of all the colors in a particularimage is performed, even if the image samples only a part of the devicegamut.

Referring to FIG. 19, steps 1804-1807 of FIG. 18A are shown. Aspecialized verification image is analyzed either with a SOM 13 or animagical 14 according to the following procedures: Step 1: Render animage consisting of homogeneous samples (“patches”) of three types: a)patches whose nominal colorant specifications match those of patches inthe original calibration, b) patches whose nominal specs are differentand c) patches specified as color. The original calibration procedure(see FIG. 8) produced statistical estimates of within-sheet andbetween-sheet color differences or process variation. User-definedrequirements for accuracy are expressed in terms of standard deviations(or like quantity) within the process variation to define confidencelimits for the process. Three kinds of color error derived fromverification procedures are used to control the process and are referredto the confidence interval in order to decide if recalibration isnecessary.

Step 2: New measurements of type “a” patches (preceding paragraph) arecompared to historical values to estimate change in the process; athorough sampling of color space is useful because it is possible thatchange is not uniform with color. Step 3: Model predictions of thecolors of patches of types “a” and “b” are compared to measurements toyield estimates of the maximum and average values of color error for theforward model. Step 4: Comparisons of the requested colors of patches oftype “c” to those obtained (measured) are used to estimate the overallerror (due to change of the process, model error, model inversion error,interpolation and quantization errors, when relevant.) Step 5: If colorerrors assessed in this way exceed the confidence limits, the User(s)are advised that the system needs recalibration and corrective actionsmay also be taken, such as modification of conditioning transforms,depending on the severity of the problem. If the device is a press,color error data is furnished to the press control system (subject toUser intervention) which does its best to bring the press as close tocriterion as possible.

The advantage of specialized imagery is that suitably chosen patchesprovide more information than may be available from imaging colorimetryof images with arbitrary color content. This can guarantee that theentire gamut is adequately sampled and can provide differentiatedinformation about sources of error. However, imaging colorimetry of thereproduction during or shortly after rendering is often the leastobtrusive way of verifying that forward models and renderingtransformations are current, especially when a volume production deviceis the object.

Because, as was noted above, the variations in color need not be uniformthroughout the gamut of the device, the data structure is segmented intoclusters of contiguous cells in order to identify the most frequentcolors in the various regions of the gamut. Thus, system 100 hereinsamples color errors throughout the image. The processing checks to makesure that the frequency it is reporting for a cluster of cells is apeak, not a slope from a neighboring cluster.

In order to improve the reliability with which corresponding peaks indifferent histograms can be resolved, methods of image processing suchas accumulation of counts from multiple images (averaging,) bandpassfiltering and thresholding are employed. Then regions of the histogramsare cross-correlated. Cross-correlation is a technique discussed in manytexts of signal and image processing, in which two functions areconvolved without reflection of one of the two. It is similar totechniques in the literature of W. K. Pratt, Digital Image Processing,N.Y.: Wiley, 1978, ch. 19, pp. 551-558. A “cross-correlogram” revealsthe offsets of one histogram with respect to another in three-space.

The color offsets of the peaks are expressed as color errors. These aremade available for numerical printout as well as for visualization. Inthe latter, the user may choose to view a monochrome version of theimage overlaid with translucent color vectors showing the direction andmagnitude of color-errors, or may choose to view a simulation of theerror in a split screen, full color rendering of two versions of theimage data showing what the error can be expected to look like.

For clarity, an equivalent procedure for cross-correlation can beoutlined as follows: 1) subdivide the histograms into blocks and“window” them appropriately, 2) calculate Fourier Transforms of thehistograms, 3) multiply one by the complex conjugate of the other, 4)Inverse Fourier Transform the product from 3 and 5) locate the maximumvalue to find the shift in the subregion of color space represented bythe block.

For the simplest level of control, the inverse of the color errors maybe used to prepare a conditioning transformation which then modifies therendering transformation employed in making another proof. For moresophisticated, on-line control, the data are used to compute errorgradients of the sort described earlier and used by optimization anderror minimization algorithms. Results are fed to the control processorof a press or used to modify the rendering transform as a controlmechanism for a press (or press-plate) which does not use a pressbearing fixed information.

The goal is to determine errors in color reproduction in aresolution-independent way. This is shown in reference to FIG. 20,illustrating processes 1804-1807 in FIG. 18A when an imagical 14 isverifying using live image data. In step 1 of FIG. 20, the histogram isdefined. Generally, it is a data structure addressed similarly to theConditioning Transform (color-to-color' table) described earlier,although the range of colors represented in each dimension of thestructure is adaptive and may depend on the imagery. In step 2, the 3-Darrays to hold accumulated histogram data are allocated in memory andinitialized; one array is needed for “live” image data and the other forreference data. At step 3, capture of “live” image data occurs. Opticallow-pass filtering may precede image capture, preferably by a solidstate electronic sensor, in order to reduce aliasing in signalprocessing. The electronic, pixel data are converted into CIEcoordinates, and, simultaneously, a histogram of the relative frequencyof occurrence of colors in the image is stored. As mentioned earlier,the data structure may be segmented into clusters of contiguous cells inorder to identify the most frequent colors in the various regions of thegamut.

In part 4 of the process, the image data (not that captured by theimagical, but the “original,” image data) are processed through thecolor-to-color' transform to produce Reference Color Data which areaccumulated in a histogram structure. It is important to recognize whatthe requested (or “reference”) colors are. They are the colors(preferably in CIE Uniform coordinates) which are the final outputs ofall the color-to-color' conditioning transforms (usually exclusive ofcolor-coordinate transforms) and thus represent the interpretation ofthe image data negotiated by the Users.

In steps 5 and 6, as described above, the program checks to make surethat the frequency it is reporting for a cluster of cells is a peak, nota slope from a neighboring cluster. Accumulation of counts from multipleimages (averaging,) bandpass filtering of histograms, thresholding,autocorrelation and other operations of image processing are used toimprove reliability and the ability to resolve peaks and matchcorresponding peaks in different histograms. Ordered lists of peaks areprepared and written to the shareable portion of the VP. The lists arecompared and corresponding peaks identified. The color offsets of thepeaks are expressed as color errors. In step 7, color error data aremade available to User/Operator and control systems.

Referring to FIGS. 21A-21F, a Graphical User Interface (GUI) to theapplication software is shown. The GUI is part of the software operatingat the nodes in network 11 for conveying the workings of system 100 at ahigh-level to the user. The user-interface has reusable softwarecomponents (i.e., objects) that can be configured by users in apoint-and-click interface to suit their workflow using establishedvisual programming techniques. The GUI has three functions: 1) Network(configure and access resources,) 2) Define (Transformation) and 3)Apply (Transformation.) All three interact. For instance, verificationfunctions fit logically within Apply Transformation but must be able tofeed back corrective prescriptions to Define Transformation whichprovides a superstructure for modules concerned with calibration andmodeling. Therefore, both Define and Apply need access to ColorMeasurement modules, whether they make use of imaging or non-imaginginstruments. “Network” is responsible for coordinating network protocolsand polling remote nodes. Part of this function includes theidentification of color measurement capabilities of a node. Another partis to insure that a user's mapping of software configuration onto hisworkflow is realizable. Loading the appropriate color measurement devicedrivers is as crucial as choosing and initializing the correctcommunications protocol and proofer device drivers. Therefore, ColorMeasurement coexists with modules for administering network protocols,building device models, building color transformations and implementingthe transformations for the conversion of color images.

For the purposes of this discussion, assume that the application isstand alone. Today, Graphical User Interfaces can be prepared viaautomatic code generation based upon re-usable components in most of thewindowing environments on the market. The depictions of menus andattributes of the User Interface in what follows are not meant torestrict the scope of the invention and are kept simple for clarity.

Referring to FIG. 21A, a first level hierarchy screen is shown allowinga user to enable configuration of network 11 nodes, remote conferencing,and user oversight of processes in system 100. Upon invocation, theapplication introduces itself and offers a typical menu bar with fivechoices (command names), File, Network, Define, Apply and Help. ClickingFile opens a pull-down menu whose selections are like those indicated by221 in the FIG. Clicking on Create 222 initializes file creationmachinery and opens the Network tableau (FIG. 21B) in a mode ofreadiness to design a virtual proofing network. Export VP (VirtualProof) 223 offers the option of converting color transformationalcomponents of the Virtual Proof into standardized file formats such asInternational Color Consortium Profiles, Adobe Photoshop colorconversion tables, PostScript Color Rendering Dictionaries, TIFF orTIFFIT. A possibility of importing standardized color transformationfiles is also provided. Other menu items under File are conventional.

The Network heading 224 opens a tableau concerned with membership in thevirtual proofing network, the physical and logical connections amongnodes and the equipment at the nodes and its capabilities. The Defineheading 225 provides means of calibrating (characterizing) devices,enabling customized assemblies of procedures to be applied toappropriate target combinations of devices and color measurementinstrumentation. The Apply heading 226 covers display of imageryrendered through the virtual proof and provides tools for verifying andreporting on the accuracy of color transformations, customizing thosetransformations, conferencing, comparing various versions of the proofand establishing contact with other applications performing likefunctions. The Main menu offers Help and each of the Network, Define andApply menus offer Tutorial interaction.

Clicking on Network in the Main menu opens a tableau of FIG. 21B, whichis concerned with Connections and Capabilities. Clicking Connection 227reveals a sidebar concerned with tools and attributes of network 11connections. For example, during creation (see FIG. 1,) it is possibleto pick up a wire by way of the “wiring” entry of the sidebar 228 andmove it into the field as part of assembling a network model that can bewritten to a file and can direct proofing commerce. Double clicking onthe wire reveals information about the connection or permits editing ofthe information when the “Modify” radio button 229 is activated. Errornotices are generated when the software drivers required to implementthe model are not available or when the hardware they require is notpresent. The present invention is not wedded to particular current(e.g., modem, ISDN, T1, satellite, SMDS) or anticipated (ATM)telecommunications technologies, nor to particular networking protocols.Nodes can be defined, given addresses, security, etc. and can beequipped with proofing devices in a manner similar to the design ofconnections. The summary of the network's connections and of nodalcapabilities is shared through the Virtual Proofs tagged file formatdescribed earlier which is kept current at all sites.

In the center of FIG. 21B is an example network topology. It resemblesthe network of FIG. 1, where “cli” 230 refers to a possible client(e.g., advertiser) member, “ad” 231 an ad agency, “pub” 232 a publisher,“eng” 233 an engraver and the Ps are printers. The links between thenodes are created and modified through the wiring 228 functionality of“Connection” mentioned above. Clicking “Capability” reveals a sidebar234 concerned with devices and their associated instrumentation forcolor calibration and verification. An example of the use of the menu isas follows: I am a user at an ad agency who wants to establish a remoteproofing conference regarding a page of advertising with a publisher andprinter. I bring up the relevant network using “Modify . . . ” in FIG.21A, push the radio button for view/select 235 in FIG. 21B, and click onthe “pub” node 232. This creates a connection between the ad and pubnodes if one can be made and initiates a process of updating virtualproof files at either end of the link. Then I click on hard proof 236and color measure 237. This utilizes the updated VP information to showme what hard copy proofer(s) are available and how they are calibratedand/or verified. Then I follow a similar sequence of actions withrespect to a P node. The initiation of display and conferencing aboutcolor image data is done via the Apply menu of FIG. 21D.

To pursue the example of the preceding paragraph, suppose that I findthat the hard copy proofer at node “pub” has not been calibratedrecently. A study of the information about the device in the updatedVirtual Proof reveals whether re-calibration or verification procedurescan be carried out without operator intervention at that site. From onesite or the other, the Define (Transformation) menu of FIG. 21C providesaccess to the tools and procedures for calibrating while the Apply(Transformation) menu (FIG. 21D) supports verification. A node can beactivated in the Network menu and then a device at the node singled outfor calibration within Define.

Clicking on “Node” 241 in the bar atop the Define menu of FIG. 21C opensa pull down providing access to other nodes without requiring a returnto the Network menu. Which node is active is indicated at the upper leftof the menu 242. Clicking on “Devices” 243 opens a pull-down which listsclasses of devices; clicking on a member of that list displays aninventory of devices of that class present at the active node. Selectinga device in this list is the first step in the process of calibrationand causes the device to be identified as active 248 at the top of themenu. The classes of devices of particular interest in the invention areimaging colorimeters or imagicals 14 (“imagicals,” 244,) unitarycolorimeters (SOMs 13, capable of measuring discrete tint samples, 245,)presses 246 and proofers 247 of hard and soft copy varieties. Clickingon “Procedures” 249 reveals a pull down listing calibration modules 250such as linearization, Forward Model Generation, etc.

Procedures appropriate to the device can be dragged and dropped into theopen field at the center of the menu and linked by connecting arrows.The controlling application software monitors the selections, performserror-checking (with notification and invocation of tutorial material)and links together the modules needed to perform the task if possible.FIG. 21C shows a flowchart for complete calibration of a renderingdevice encircled by a dotted line 251. In the case of some proofingdevices, such as Cathode Ray Tube displays and Dye Sublimation printers,it may be sufficient to perform complete calibration only infrequently.In particular, it is usually adequate to re-compensate for the gammafunction of a monitor (a process which falls under “linearization”) on aregular basis. Because phosphor chromaticities change very gradually,re-determination of the color mixing functions of the calibration neednot be performed as frequently. Therefore, a user may activate the CRTdevice at his node and specify only linearization in preparation for aproofing conference. Alternatively, the present invention covers theequipment of monitors with devices that can participate in continualverification and recalibration.

The “Apply” Transformation menu (FIG. 21D) provides access to thedatabase of pages and images that are the subject of remote proofingthrough the “Page/Image” 256 selection in the menu bar. Clicking heredisplays a shared file structure. Although the (generally) bulky colorimage data file of interest need not be present in local storage 19 ofFIG. 1 at all nodes where proofing is to occur, it is generallydesirable to make a local copy so that rendering across the network isnot necessary. However, one of the purposes of the Virtual Proof is tomake multiple transfers of the bulky data unnecessary. The “Node” 257and “Devices” 258 elements of the menu bar have effects that areentirely analogous to those in the “Define” menu of FIG. 21C. More thanone device at a node can be made active in support of a mode in whichinteractive annotation and conferencing via the medium of a soft proofon a video display is employed to negotiate changes in a hardcopy,remote proof that is taken to be representative of the ultimate clientdevice.

Clicking on “Procedures” 259 in the Apply menu bar of FIG. 21D reveals apull down that includes functions such as “Render to display . . . ”260, “Verify . . . ” 261 and “Window . . . ” 262 to externalapplications. Rendering supports display, either within the Apply windowor on a separate, dedicated video display, of imagery a) as the designerimagined it, to the extent that the proofer is capable of showing it, b)as a client device, such as a press, can reproduce it and c) as anotherproofer is capable of representing the press, including indications ofgamut mismatches superimposed on the imagery and location of errorsidentified by the verification process. To further the example, thevirtual proof may mediate a rendering of an image as it appears or willappear on press. If the node includes an imaging colorimeter, then animage of the proof can be captured and analyzed in order to provideverification of how well the proofer represents the client. Withoutverification, digital proofing and remote proofing for color approvalare not really useful.

The Apply menu provides a Window 262 through which to “plug-in” to or to“Xtend” applications such as Adobe Photoshop or Quark Xpress. It alsoprovides the platform for incorporating remote, interactive annotationof the sort provided by Group Logic's imagexpo, reviewed earlier.Imagexpo focuses on marking up images with a virtual grease pencil, aconcept that is extended in the present invention to remote conferencingconcerning gamut scaling, black utilization and other aspects of thedefinition of rendering transforms. Aspects of rendering such as blackutilization 263 (or gamut operations) can be harmonized across theproduction network by sharing/exchanging black designs through thevirtual proof file structure and mechanism.

Menus supporting interactive design and selection of user preferencedata are shown in FIGS. 21E and 21F. A User-Interface to supportinteractive determination of black utilization is depicted in FIG. 21E.It may be invoked from either Define or Apply menus. At the top right270 is shown a panel of colorant specifications which are to beconsidered neutral in color. Users may redefine entries in the panel byclicking and keying or by modifying curves in the graph below 272provided the graph is toggled to neutral definition rather than GCR. Inthe neutral definition mode the user may move points on any of thecolorant functions; the points are splined together and changes in thegraph are reciprocal with changes in the panel. A “return to default”switch 274 provides an easy way to get out of trouble. At the upperright 276, 278, variable readout switches enable definition of maximumcolorant coverage and maximum black. At the bottom right 280, “CustomizeTonal Transfer” opens the door to user modification of one or more ofthe 1-dimensional output postconditioning LUTs which are part of thecolor to colorant transformation. The application warns sternly that thespecification of transfer curves which did not prevail duringcalibration will void the calibration; however, there are situations inwhich knowledgeable users can make effective use of the flexibilityafforded by this function.

When the Graph is switched 282 to GCR mode, the user can control onlythe shape of the neutral colorant curve; because GCR is colorimetric,the non-neutral curves respond compensatorily to changes in black. Therelationship of the curve specified in the graph to the amount of blackchosen for a solution at a particular entry in the separation table isas follows: The function shown in the graph represents amounts of thecolorants which produce given neutral densities. At each density, thereis a range of possible black solutions from minimum to maximum. At theminimum, black is zero or one or more of the non-neutral colorants ismaxed out; at the maximum, black is at its limit and/or one or morenon-neutral colorants are at their minimum. In this invention, the % GCRis the percentage of the difference between min and max black chosen fora solution. By industry convention, a constant % GCR is seldom desiredfor all Lightness (density) levels. Therefore, the black curve in thegraph defines the % GCR desired as a function of density. Although it isconceivable to make GCR a function of hue angle and Chroma as well as ofLightness, this is usually an unwarranted complexity with one exception:It is useful to graduate GCR with Chroma when preparing transformationsfor more-than-four colorants as discussed earlier. This need isaddressed through the “GCR special . . . ” submenu 284 offered at thebottom right of FIG. 21E.

FIG. 21F depicts a GUI screen to gamut operations. Clicking on “Gamuts”reveals a pull-down 286 which provides access to lists of input, outputand client gamuts; the latter two are both output gamuts, but a clientgamut is known to the system as one that is to be represented on anotherdevice. It is possible to drag and drop members from the different typesinto the field of the menu and to link them with various procedures, aswas the case in FIG. 21C. Procedures applicable to the gamutsinclude: 1) Analyze 288 which displays information on how a particularconditioning transformation was put together (e.g., was it aconcatenation of gamut scaling and color aliasing operations?—whichones?) and on gamut compression records, a constituent of the VP whichstores key variables of a gamut scaling, such as minimum Lightness,cushion value, functional form, etc. 2) Apply to Imagery 290 enablesdisplay of imagery mediated by transformations configured in this orrelated menus on some device. 3) Compare Gamuts 292 enablesvisualization, in several forms, of the relationship between the gamutsof two or more devices—this functionality is elaborated in a followingparagraph. 4) Concatenate 294 does not apply solely to gamut operations;it links nested or sequential transformations into implicit, nettransformations. 5) Gamut Operator 296 provides a graphical display ofan operator; this is a different representation of the informationavailable from Analyze 288. 6) Negotiate Proofing Relationship 298 worksclosely with Compare Gamuts 292; it enables users to make decisionsbased on information provided by Compare, such as whether to use theLeast Common Gamut as the aim point for a network of volume productiondevices. 7) Remap Hues 300 provides the separable hue adjustmentfunctionality described earlier. 8) Rendering intents 302 is a mechanismfor providing users with generic gamut scaling options for accomplishingthings like obtaining the most saturated rendering on paper of colororiginally created in a business graphics application on a videodisplay. Compare Gamuts 292 allows a user to invoke and control the useof gamut filters, which were described earlier in connection with FIG.17.

System 100 supports the coordination of color control in multisiteproduction in several forms. Because the virtual proof encompassesrecords of the states of calibration of network devices independent ofthe color data, application software can define a criterion forreproduction across the network in one of several ways. Based on thestates and capabilities of network devices, a criterion may be selectedwhich all devices can satisfy, such as a least common gamut (LCG). LCGdefines the aim point for production and the control system strives tominimize the color error with respect to it. Alternatively, users mayselect one device as the criterion and all others are driven by thecontrols to match it as closely as possible. Optionally, users maychoose to disqualify one or more rendering devices on the networkbecause it/they cannot match the criterion closely enough or due tofailures of verification.

Referring to FIG. 22, an example of system 100 (FIG. 1) is shown havingtwo nodes of the network. Node N may possess high performance computingprocessor(s) and, optionally, extensive electronic storage facilities.Node N may also have output devices of various types along with colormeasurement instrumentation for the calibration of those devices and itmay be connected to more than one networks for Virtual Proofing.

In addition to participating in one or more networks for VirtualProofing, Node N may assist other nodes by computing transformationfunctions. It may also function as a diagnostic and service center forthe networks it supports.

Node A 310 includes components similar to nodes 102 or 104 (FIG. 1) andis linked to other nodes by network link 11 a. The communication betweenNode A and Node N is enabled via the Internet or World Wide Web, forexample, to a web site service “cyberchrome” 315 at Node N 312. Thiscommunication is illustrated by the screen of the video display device311 being labeled “www.cyberchome”. Node A 310 has a computer 314 suchas a personal computer or workstation in accordance with applicationsoftware providing Virtual Proofing as described earlier. Node A 310need not have a computer other than the processors embedded in theproofing devices or color measurement instrumentation. In this case, theVirtual Proof for Node A are coordinated by another computer system,such as the computer server 316 at Node N (called hereinafter theprofile server), as described earlier in connection with FIG. 1.

Printer 317 is shown for Node A, but not for Node N. A color measurementinstrument (CMI) 318 is provided as a module for calibration of theprinter. CMI 318 includes a sensor, lamp, reference and control unit(which may itself be of modular design) and a transport mechanism 320for transporting hard copy of a calibration or verification sheetrendered by printer 317 so that the sheet may be read by the CMI with aminimum of user effort or involvement. Reflection or transmissionmeasurements are facilitated by the transport mechanism 320 for such asheet, bearing a matrix or array of color samples, which is actuated byclick of computer mouse or, preferably, by insertion of the sheet in thetransport mechanism. The transport mechanism may be integrated with theprinter 317 in which the optical pickup component of the sensor ismounted to move in tandem with the marking head of the device andtransport of the copy may be performed by the mechanism of the printer,such as when the printer is an ink jet printer. In either case, theoptical pickup link their devices to a control unit for the CMI by fiberoptic or by electrical wire link.

The profile server 316 at Node N 312 may consist of a multiprocessor orlocally networked array of processors or high performance workstationprocessor whose performance may be enhanced by special-purpose hardware.The exact architecture (such as RISC or CISC, MIMED or SIMD) is notcritical, but needs to provide the capacity to compute quickly colortransformation mappings, gamut operations, etc., as described earlier.Any of the processors in the network may have this capability or nonemay. However, the more responsive the network is in development andmodification of Virtual Proof constituents, the more useful it can be.Disk storage or memory 321 (similar to storage 19 in FIG. 1) representscentralized storage of current and historical constituents, which may beshared by one or more nodes on the network. The profile server furtherprovides a database which stores calibration data for rendering devicesof the network, such as color profiles (inter-device color translationfiles), or data needed to generate such profiles. The calibration dataproduced for each rendering device in the network was described earlier.

Referring to FIGS. 23-32, the calibration of video color monitors ordisplays shown in FIGS. 1 and 2 will be further described. Videodisplays play an increasingly important role in color communication.They are used to simulate three-dimensional rendering in synthetic imagecreation, as such they function as linear, 3-channel, input devices, andare also used ubiquitously in interactive image editing. Further, videodisplays can be used as soft-proofing devices, thereby providing a videoproofer. In the latter application, it is desired to portray an image ona video proofer as it will be rendered on a hard copy device. Althoughit is not usually possible to match the spatial resolution of hard copywith a soft proof, it is possible to forecast color reproduction.

Although the following describes color calibration of cathode ray tube(CRT) displays, it can also be applied to any video display technology,such as Digital Light Processing (based on Texas Instruments DigitalMicromirror Device), flat plasma panels, Liquid Crystal Display panels,etc. The foregoing technologies may be used in front or rear projectionapplications. An embodiment in which a front projection device is usedto project high resolution imagery onto production paper stock wasdescribed earlier.

Video displays are highly complicated image reproduction devices,featuring ample adjacency effects (interactions among neighboringpixels, where a neighbor may be spatially removed at some distance.)However, satisfactory results may be obtained by ignoring complexspatial effects and modeling three, separable variables simply: 1) colormixture, 2) gamma or the intensity-voltage relationship and 3) thedependence of luminous output at a pixel on where it is on the screen,independent of its interactions with the level of activity of otherpixels.

All video displays behave as light sources and those suited to accuratecolor reproduction conform to linear rules of color mixture. In otherwords, they observe, to a reasonable approximation, the principle ofsuperposition. This means that the light measured when all threechannels are driven to specified levels equals the sum of the lightmeasured when each is driven separately to the specified level. Also,the spectral emission curve (indicating light output as a function ofthe wavelength of the light) changes by a constant scale factor as thedriving voltage changes.

The last point is illustrated for green phosphor emission in FIG. 23.The emission spectrum is a property of the phosphor and is normallyinvariant during the useful life of the CRT. The graph shows theactivity in the green channel, as a function of wavelength, when drivenfull scale at digital level 255, denoted by numeral 322, compared with15 times the activity in response to digital level 64, denoted bynumeral 323. The fact that the two curves superimpose means that theydiffer by a factor which does not depend on wavelength—a linear propertyimportant for color mixture. However, the scale factors are not linearwith digital drive levels, a manifestation of non-linear gamma in thedevice. The derivation of color mixing transformation matrices capableof translating RGB device codes for a particular monitor into XYZTriStimulus Values or vice versa is described, for example, in Holub,Kearsley and Pearson, “Color Systems Calibration for Graphic Arts. II:Output Devices,” J. Imag. Technol., Vol. 14, pp. 53-60, April 1988, andin R. Berns et al., “CRT Colorimetry, Part I Theory and Practices, PartII Metrology,” Color Research and Application, Vol. 18, Part I pp.299-314 and Part II pp. 315-325, October 1993.

It has been observed that the spectral emission functions of CRTphosphors do not change over significant periods of time. This meansthat for a large class of displays, the data needed for color mixturemodeling can be measured once, preferably at the factory, and relied onfor most of the useful life of the monitor. Phosphor spectra and/orchromaticities measured at the factory may be stored in Read Only Memory(ROM) in the display, they may be written on a disk and shipped with themonitor or, preferably, they may be stored in association with themonitor's serial number in a “Cyberchrome” database at Node N 312 ofFIG. 22. It is preferable if spectral data are captured and stored. Inthe database, they can be made accessible to networks for VirtualProofing via restricted, keyed access, using typical encryption andsecurity schemes for the Internet.

The balance of Red, Green and Blue channels does vary continually, oftenrequiring adjustment at least daily. RGB balance determines the whitepoint. Key variables subject to change in the R, G and B channels arethe bias and gain. The bias, or offset, determines the activity in thechannel at very low levels of input from the host computer system, whilethe gain determines the rate of increase in activity as digital driveincreases and the maximum output. Bias and gain controls usuallyinteract within a color channel, but should not interact betweenchannels if the superposition condition is to be satisfied. The gamma ofa channel is the slope of the relationship giving log light intensity,usually in units of CIE Luminance, or Y, and log digital drive suppliedby the host computer system. In order to maintain a given white balanceand overall stability of tone and color reproduction, it is necessary toregulate the bias and gain in the three channels. Consistency of gammais the main determinant of tone reproduction, while white pointstability depends on the balance of activity in the three channels.

Ambient illumination refers to light that reflects off the faceplate (orscreen) of the display and whose sources are in the surroundingenvironment. Light so reflected adds to light emitted by the display. Aviewer may not be able to distinguish the sources. At low to moderatelevels, ambient illumination affects mostly the dark point of thedisplay causing a loss of perceptible shadow detail and, possibly, achange in shadow colors. In effect, gamma (and tone reproduction) aremodified, along with gray balance in the shadows. Generally, it ispreferable to detect and alert to the presence of ambient contaminationso that users can control it than to incorporate ambient influences intothe calibration.

The foregoing several paragraphs are meant to set the stage fordiscussion of monitor calibration in the color imaging system of thepresent invention.

It is often very difficult to build a good colorimeter, as discussed inthe article by R. Holub, “Colorimetry for the Masses?” Pre magazine,May/June, 1995, for video displays. Accurate, repeatable ones areexpensive; for example, one marketed by Graseby Optronics sells foraround $6000, currently, and a fine instrument by LMT costs considerablymore. FIG. 23A shows a comparison between measurement of spectralemission of a red phosphor of EBU type by a PhotoResearch PR700Spectroradiometer 324 and a Colortron II 325. The plots make clear thatan instrument capable of resolving 2 nanometer increments can resolvethe complex peaks in the phosphor's spectrum while Colortron cannot.Colortron is a low-resolution, serial spectro as discussed in theearlier cited article “Colorimetry for the Masses?”. Calculations ofTriStimulus Values and chromaticities based on the two spectra revealsignificant errors in the estimation of red chromaticity with ColortronII.

The foregoing analysis suggests that, for phosphor-based displays, atleast, continual colorimetry is not required to maintain goodcalibration. Thus, maintenance of gamma and white balance and monitoringof ambient influences is all that should be required for aphosphor-based display. Such maintenance is described later isconnection with FIG. 32.

FIG. 24 shows a configuration of color display 326 (i.e., CRT), coweland color measurement instrument, as generally shown in FIG. 2, thatenables very accurate, inexpensive and automatic maintenance of CRTdisplay calibration. Additional features of this configuration willbecome apparent from the following discussion of FIGS. 24, 24A, 24B, 24Cand 24D.

The cowel 322 is circumferential and helps to shield the faceplate 328from stray light coming from any direction, including the desktop onwhich the display 326 may be located. Cowel 322 is black-coated on innerfaces 322 a to absorb light. The cowel 322 forms a light trap 329 forthe embedded sensor 330 coupled to the cowel. A small lamp 335 islocated beneath the lower flange 322 b of the cowel. Lamp 335illuminates the desktop without influencing the display significantly incircumstances when overall room illumination is kept low for goodviewing.

A sensor 330, often referred to herein as an electro-optical pickup, islodged in the cowel. It views approximate screen center, denoted by line331, (however other areas of the screen may be viewed) collecting andfocusing light onto the sensor. The line of sight 334 reflects off thescreen 328 and into the light trap 329 formed by the lower flange of thecowel, so that specularly reflected light will not enter the sensor. Itpermits unattended calibration, possibly during screen-saver cycles orat other times when the operating system of the computer at a node isnot scheduling activities which would compete with calibration. Becausethe sensor 330 is perched in the cowel 332, it does not require userinvolvement to be affixed to the screen, it leaves no saliva or otherresidue on the faceplate, it does not contribute to desktop clutter andit is well positioned to monitor ambient illumination influences.

FIG. 24A shows an example of the assembly of cowel and sensor for acolor monitor, wherein the circumferential cowel 401, arm member 410,and one of the brackets 404 of FIG. 24A are separately shown in FIGS.24B, 24C, and 24D, respectively. To satisfy the goal of adapting thecowel to monitors of different size and make, the circumferential cowel401 is a separable part which can be sized apropos of 17, 20, 21 or 24inch or other size monitors. It can be made of any material, butpreferably is of light weight plastic. The rear edge 402 of the cowel401 rests against that part of the plastic monitor cover which framesthe faceplate of a typical monitor. This has two advantages: First, itprovides relief to the mechanism 412 which suspends and supports thecircumferential flange 401 a of the cowel 401 because a significantcomponent of the force of support is into the front of the monitorchassis. Second, the upper flange 401 b of the cowel 401 will supportthe electro-optical pickup head at socket 403 at a correct andpredictable angle of view of the screen.

The mechanism 412 for supporting circumferential cowel 401 on themonitor includes a rack and pinion adjustment mechanism having anextending arm member 410 with a knob and gear 405 to allow adjustment oftwo brackets 404 having teeth which engage opposite sides of the gear405. The brackets 404 have ends 406 which attach to the top right andleft corners of the monitor's chassis. In other words, adjustment of theknob causes two foam feet at either end 406 of the brackets to move inand out so as to control their pressure against the sides of thechassis. In other words, the sliding bracket rests atop the monitor'schassis and is held in place by an adjustable press fit to the sides ofthe chassis. It, in turn, provides additional support to thecircumferential flange 401 a. Arm member 410 connects the brackets 404to the circumferential flange 401, such as by screws coupling arm member410 and upper flange 401(b) together. Alternatively, the knob and gear405 could be replaced by a friction-fitted tube sliding within a sleeve.Cables 407 represents wires being led from the electro-optic componentsin socket 403 back to an interface such as Universal Serial Bus, of thecomputer 314 coupled to the monitor 311 (FIG. 22). However, it couldalso represent an alternate means of attachment, if it were madeadjustable front to back of the monitor's chassis. The arm member 410rests partially on the circumferential flange 401 at upper flange 401(b)and provides a housing for the electronics (circuitry) 408 associatedwith the electro-optic unit which plugs in at socket 403.

The depth, denoted by numeral 409 of the circumferential flange 401 a isthe distance from its outermost edge along a perpendicular to the pointat which it rests against the monitor's chassis. Due to contouring ofthe chassis or the cowel (in the interests of a good fit to the chassisor of aesthetics) the depth may vary around the circumference. We havechosen, for example, a depth of 8 inches, on average, to effect a tradeoff between two factors: the degree of shielding of the viewing area ofthe screen from environmental stray light (ambient), and thedesirability of enabling more than one user an unimpeded view of thescreen at one time. At 8 inches, it should be possible for 2 or 3 peopleseated in front to see the full screen and for 2 or 3 people standingbehind them also to see. However, the particulars cited here are notmeant to limit the generality of the invention.

Many users like to conduct soft proofing under circumstances in whichthe hard copy proof may be available for comparison to the soft proofvisualized on the monitor. Under said circumstances, it is important tocontrol the viewing conditions of the hard copy. A viewing hood such asthe SoftView™ made by Graphic Technologies, Inc. is often used for thispurpose. However, it is also important to insure that the viewing hooddoesn't contribute to stray light and that it is positioned tofacilitate soft- and hard-proof comparisons.

Accordingly, the viewing box (or reflection viewer or hood) 501 shown inFIG. 25 may optionally be provided at Node A 310 (FIG. 22) which isintegrated with the cowel 401. FIG. 25 shows an arrangement in which thevideo display is raised up on a pedestal with viewing box positionedbelow, so that the top side of the box is contiguous with the bottomflange 401 c of the cowel. One side of the box has an opening 605opposite the back wall 608 of the box upon which media is locatable.Other arrangements are possible, for example, some workspacearrangements may call for situating viewing hood and monitorside-by-side, i.e., integrated to the right or left side of the cowel,and oriented at an angle so as to optimize viewing at relatively shortrange.

FIG. 26 is a scale drawing, showing the configuration of light sources601, lenses 602 and reflectors 603 in the viewing box 501, whose sizeshould be adequate to accommodate at least a two-page spread inpublication terms (11×17 inches.) At top and bottom (or left and right)are ballasts 604 which can accept one or two (depending on requiredillumination levels) Daylight 5000-simulating fluorescent lamps. D5000is mentioned merely because it is a standard illuminant in certainindustries; in other industries, the choice might be different.Approximately pear-shaped plastic lenses serve to diffuse and dispersethe light. The lenses should be frosted or waffled (rippledlenticularly) so as to enhance diffusion of light and should not greatlychange the color temperature of the fluorescent light.

Any direct sighting of the lamps/diffusers by the viewer 501 (in normaluse) is prevented by reflectors 603 which are aluminized or chromed onthe side facing inward toward the copy so as to reflect and furtherdisperse the light. The configuration depicted has been shown to resultin very uniform, diffuse illumination over the area of at least a twopage spread. Angles 605 and 606 are approximately 14° and 34°,respectively. An eye 607 is depicted on a line of sight which shouldextend at least 25 inches back from the rear viewing surface 608 of theviewing box. The single lines 609 and double lines 610 indicate theextent of the illuminated back wall of the viewer which can be seen fromdistances of 40 inches and 30 inches, respectively.

The above discussed assembly is preferably equipped with a dimmer. This,along with the choice of number of fluorescent tubes, allows criticalcontrol of the brightness levels within the viewing box. Brightnessshould be made as similar to that of a full-field white on the videodisplay as possible. An additional option for the viewing box 501 is theinstallation, in the back wall 608, of a light box able to accommodate8×10 or smaller color transparencies, as is provided in otherconventional viewing hoods used in industry.

As explained previously, the primary chromaticities of phosphor-baseddisplays are very constant over time. Therefore, the sensor coupled tocowel 332 (FIG. 24) or 401 (FIG. 24A) should be a simple, calibratedluminance meter, called hereinafter lumeter, sufficient to maintainwhite balance and gamma and to detect significant increases in ambientillumination over the norm.

However, display technologies which rely on light sources, such as LCDor DMD as defined earlier, may experience drift in primarychromaticities as the external light source ages. It is common for frontor rear projection displays based on liquid crystals or digitalmicromirrors to employ metal halide lamps or xenon arcs. The spectraldistribution of the former certainly changes with time and that of thelatter may under some circumstances. In these cases, adequate control ofcolor balance requires ongoing colorimetry, such as provided by aspectral based CMI. Even with CRTs, there are occasions in whichadequate calibration or re-calibration may involve a color-capabledevice. For instance, it may be desirable to retrofit a CRT which wasnot calibrated at the factory with the invention in order to confer itsbenefits on a user. Accordingly, several types of electro-optic modulesare described below, from a Lumeter to a spectral device.

Referring to FIG. 27, a cross-section of part of the sensor integratedwith the upper flange 401 b of the cowel in socket 403 (FIG. 24A) isshown having a housing 701 including optics. On the assumption that thecenter of the outer edge of the upper flange is 7.5 to 8 inches from thechassis and that the flat surface of the flange is perpendicular to thefaceplate, the center of the hole bored for the lens tube is 6 inchesout. The bore is cocked at an angle of 53° to the flat surface of theflange so as to look at the approximate center of a nominal 21 inchscreen. For example, the length of socket 403 may be 2 inches, and 1inch in diameter.

The lens 702, shown in cross section in the tube, is a commercial gradecondenser, plano-convex with diameter of about 20 mm and focal length ofabout 25.4 mm. It actually forms an image of what is at screen centerabout 36 mm behind the lens, presumably due to the diffuse and divergentnature of the light source. The inner surface of housing 701 should bevery light absorbent. It should be painted flat black, or, preferably,outfitted with a conical baffle 704 with porous black inner surface. Theabove numerical values of components in FIG. 27 are exemplary, and othervalues may be used. In particular, a 53° angle is not quite as desirableas 45°, but the important consideration is that the line of sightreflected off the center of the faceplate intersect the black trapformed by the bottom flange of the cowel, as shown in FIG. 23. The blacktrap shields the sensor from light rays specularly reflected off thefaceplate of the monitor.

FIG. 27 also shows an inner tube 705 which slides into the outer lenstube on tracks 707. It is used to bring either a photodiode array or afiber optic 706 up to the focal plane behind the lens. The photodiodearray is employed in the simple lumeter while the fiber optic pickup isused with a spectrograph. In the lumeter, it is desirable not toposition the sensor exactly in the focal plane so as to induce a littleblurring over the sensor. In the case of the spectrograph, the fiberoptic should be chosen to have an acceptance angle which is ascompatible as possible with lens 702 and with the spectrograph to whichit is coupled.

Separate tubes may be used for the lumeter and the spectrograph. Theformer could employ a non-achromatic plastic lens whereas the latterbenefits from a glass achromat. The two kinds of tube should mate withthe flange interchangeably.

Referring to FIG. 28, a block diagram of the lumeter is shown withcontrol circuitry. The lumeter includes two parts which may be combinedor separate from each other, an electro-optic pickup head 801 and thecontrol and interface electronics circuitry (controller) 802. Theelectro-optic pickup head 801 includes a lens 702 which converges lightonto sensor 804 through spectrally selective filter 803 which must, atthe least, attenuate Infrared radiation greatly. Another sensor 805 isan optional sensor identical to sensor 804 except that it is thoroughlylight-baffled (i.e., protected from receiving any light).

Sensor 804 may be a photodiode array incorporated in a TSL 230, theprogrammable member of Texas Instruments' Light-to-Frequency converter(LTFC) family. The device integrates photosensors with an amplificationstage and a current to frequency output stage which interfaces directlyto a counter on a microcontroller. The lumeter described is verydesirable because of the high levels of sensitivity, repeatability andlow parts count.

Because its output is a pulse train, the TSL 23X device can be coupledto the control electronics 802 by a lead of up to several feet.Therefore, the device may be fitted into the inner tube with amplebaffling and positioned near focal plane. Wires are lead back to thecircuit board containing the control electronics. In this way, theelectro-optical pickup head 801 can be very compact. As stated earlier,the control electronics 802 may be located in a housing or cavity 408(FIG. 24A) in arm member 410 connecting the brackets 404 to thecircumferential flange. Such control electronics may also be located ina separate circuit enclosure.

Circuitry 802 consists of the control and interface electronics,including a microcontroller 807 having an on-board hardware counterwhose overflows are counted in software. This enables long lightintegration times. A multiplexer 806 of circuitry 802 selects betweenthe two sensors 804 and 805 when both are available, while timing iscontrolled by a clock oscillator 808. A level shifter 809 provides aninterface for adjusting the output of the microcontroller for RS232,RS422, or other communication protocol, such interface may be a USB(Universal Serial Bus) interface.

The lumeter operates by cumulating pulses from the ‘23XLight-to-Frequency Converter (LTFC) so as to perform A/D conversion byintegration. It is best to use some means to ascertain the refreshfrequency of the display and to set the integration time to be anintegral multiple of the refresh period. One means is to read back therefresh frequency from the processor in the display. Another is tomeasure it with the lumeter, taking advantage of Fast Fourier Transformalgorithms and techniques, such as described, for example, inapplication notes published by Texas Instruments for the TSL 23X.However, it is also acceptable to set an integration time such thaterrors due to incomplete refresh cycles are small. An integration timeof 5 seconds satisfies this criterion for a 75 Hz refresh rate.

FIG. 29 shows a basic command set used by the host computer tocommunicate with the lumeter. For example, the series of ascii strings“<E1>” “<P0>” “<D0>” “<S2>” “<T5F5>” is used to initialize the lumeterfor data collection, where the ascii strings are enclosed in quotationmarks. The effects are to turn on echoing from the lumeter back to thehost, to power up the sensor so that it stays “awake” once a sensitivitycommand is issued, to program frequency division to a factor of 1, toset sensitivity to 100, i.e., use all available photosensors, and to setthe integration time to 5 seconds However, in production, it ispreferable for most applications to use a non-programmable version ofthe LTFC such as the TSL 235.

Intensity/Voltage data collected with a reference spectroradiometer (aPhotoResearch PR700) and with a lumeter are assembled in Table I below.These are the data needed to calculate gamma. Data are presented forseveral conditions which test the generality and robustness of thedesign. FIG. 30 exemplifies the summary data analysis. The actual points1002 come from the fourth and fifth columns of the second data set,namely full screen green at 23° C. ambient temperature. The values ofgamma quoted in what follows are the slopes of straight lines such as1001.

TABLE I Measurement of Intensity/Voltage Characteristic for SeveralConditions Digital Lumeter PR700 Log Log Log Code Counts LuminanceDigital Lumeter Luminance Measured for 6″ by 6″ square centered inscreen, green channel 15 10 0.0398 1.1761 1 −1.3997 31 65 0.3408 1.49141.8129 −0.4675 64 350 2.191 1.8062 2.5441 0.3406 128 1349 9.552 2.10723.13 0.9801 255(max) 4503 33.73 2.4065 3.6535 1.528 Measured from fullscreen, green channel, room temperature of 23 C. 15 18 0.0397 1.17611.2553 −1.4008 31 132 0.3532 1.4914 2.1206 −0.452 64 699 2.185 1.80622.8445 0.3395 128 2627 9.401 2.1072 3.4195 0.9732 255(max) 8537 32.632.4065 3.9313 1.5136 Measured from full screen, green channel, roomtemperature of 30 C. 15 23 0.0495 1.1761 1.3617 −1.3057 31 144 0.38831.4914 2.1584 −0.4108 64 732 2.311 1.8062 2.8645 0.3638 128 2696 9.6422.1072 3.4307 0.9842 255(max) 8656 33.05 2.4065 3.9373 1.5192

Gammas calculated in the manner of FIG. 30 for the various conditionsare:

-   -   a) for the Lumeter, 2.15, 2.17 and 2.08 for the 6×6 square at 23        C, full field at 23 C and full field at 30 C, and    -   b) for the PR700, 2.37, 2.36 and 2.29 for the same conditions,        respectively.        Lumeter-derived gammas are a fixed percentage of PR700 values,        indicating a robust calibration strategy and good immunity from        field size. Both instruments appear to have good temperature        compensation; however, there is about a 5% loss of gamma at the        higher temperature. This is due to increase in dark current at        the higher temperature. For applications in which higher        temperatures may be encountered and accuracy to better than 1%        is desired, a second, light-baffled sensor (805 in FIG. 28)        should be added. Adding a temperature-sensing detector is less        expensive than shuttering the system, however a shutter for        sensor 804 could alternatively be used. Because sensor 805 is        baffled, its readings indicate the dark current which can be        calibrated against temperature. Background temperature readings        can be taken with long integration cycles to insure accuracy        when the instrument is not being used for video display        calibration.

The gammas discussed above are not identical to those of a trueluminance meter because the lumeter does not have the photopicsensitivity function. However, it is preferable to save the spectralemission characteristics of each phosphor channel, as noted earlier, andto measure the spectral sensitivity function of each lumeter as part ofthe calibration process. In that way, it will be possible to calculateexactly the relations between true luminosity data and lumeter responsesfor any phosphor set and any actual lumeter. This is done by convolving(with a shift parameter of zero) the spectral emission function with thehuman sensitivity function and with the lumeter's spectral sensitivity.In this we are helped by the linear scaling of spectral emission withdrive voltage. However, if spectral data on phosphors is not available,very good calibrations will be possible based upon generic data.

The lumeter provides reproducible results from day to day, and is atleast as stable as typical reference light sources, given a suitablechoice of spectral filtering and a stable lens material. Therefore, theonly aspect of self-calibration which is of concern is the monitoring oftemperature-dependent dark signal. This is accommodated in the inventionso that the lumeter is self-calibrating, automatic and non-contact, aswill be described later below.

Referring to FIG. 31, the high level operation of the system forsoft-proofing is shown. Users want to be able to edit or retouch imagedata while seeing the images they are preparing on the video display asthey will appear on hard copy to the greatest degree possible, such asdescribed, for example, in the article by Holub, et. al., J. Imag.Technol., Vol. 14, p. 53. Automatic device calibration at a singleinstallation or coordinated among multiple nodes is provided by theabove discussed lumeter and cowel assembly.

Data generally come from one of two sources in a soft-proofingapplication, CMYK data 1101, or as 3-dimensional color data 1102,usually in a device-independent coordinate system such as CIELAB orcalibrated RGB. The first step is to get the data into the form of 3-Dcolor' data 1103, where color' means that all the colors are printable.Since CMYK data are printable by definition, they need only betranslated to suitable 3-D coordinates, as described earlier inconnection with FIG. 4A. To effect the conversion for data that startout in a device-independent notation, they must be processed through agamut operator. The latter is based upon gamut descriptors derived frommodels of the CMYK and display devices which was also described earlier.

Finally, the 3-D color' data must be translated into display-specificRGB signals. The result is accurate color output. Literal colorimetricreproduction on the video display may not be possible, such as describedin the article Holub, et al., op. cit.) The color translator 1104 mayinvolve processing beyond that needed to convert device-independentcolor' data into calibrated RGB for the particular display. However, thelatter conversion is a critical part of the translator, and the concernin this discussion is to show how it is formed or updated automatically.

Referring to FIG. 32, a flowchart of the procedures for using thelumeter to set up a video display for soft-proofing and for remoteproofing. It assumes that the color mixture component of the devicecharacterization problem has been dealt with. For simplicity, assumethat the chromaticity data derived from factory measurements of thedisplay's R, G and B channels are combined with data about the desiredwhite point to form a 3×3 matrix. The color translator 1104 thenconsists of a stage which converts 3-D color' data into XYZ TriStimulusValues, if necessary, which are then converted into linear RGB by thematrix. Resulting R, G and B values are processed through ToneReproduction Curves (“TRCs” in the parlance of the International ColorConsortium's Profile Format Specification.)

Whatever the sequence of processing steps implicit in a colortranslator, they will only be effective if the parameters of processingcorrespond to the actual state of the device. In other words, thechromaticities used to compute the matrix must be those of the device,the three channels must be balanced to the correct white point and theTRCs must compensate for the non-linear gamma of the device. Insuringthat the device is in a targeted state called calibration is the objectof FIG. 32.

State 1201 shows the Lumeter in an initial state of periodic dark signalreading. This state may be interrupted by initiative of an operator oron cue from the operating system as described earlier. Anautocalibration cycle begins by darkening the screen 1202 and measuringthe sum of light emitted by the screen (the dark emittance) andreflected off the screen by environmental sources not baffled by thecowel. Call the sum dark light.

One of the shareable components of the Virtual Proof is a default ornegotiated tolerance for dark light. If this is exceeded, credible softproofing at a given node and collaborative proofing between remote nodesmay not be possible. The device driver for the lumeter should flag theevent 1203. During a prolonged period of system inactivity, it may raisethe flag repeatedly; when an operator returns to use the system,however, he/she should be advised that an unattended calibration cyclewas not possible. There are two main ways of recovery: a) restore darklight to an acceptable level (operator action), such as adjusting theamount of ambient light reflected from the screen, or b) use a morecomplicated model of color mixture, if possible, which includes theeffects of the dark light.

It should be noted that the lumeter cannot discern the color of darklight. The latter may be important in some applications, either becauseof fairly high levels of dark light or due to a need for very criticalcolor judgments. When measurement of the color of dark light isnecessary, replace the lumeter with a spectrograph, as described laterin connection with FIG. 33.

Once the question of dark light is resolved, neutral balance 1204 isestablished. The aim white point is used in a computation of therelative activity levels needed in R, G and B channels to achieve thetarget at highlight and in the shadow. First the gains of the amplifierswithin the color display which control electron density as a function ofexternal drive are adjusted in each channel until a criterion balance isrealized. This establishes highlight white. Next, the biases of thosesame amplifiers is adjusted until the appropriate balance is realized inthe shadows. This is likely to upset the highlight balance. Therefore,several iterations may be needed to balance both highlight and shadow.

Under the simplest model of color mixture, the computation of relativeactivity levels proceeds as follows: From the factory-suppliedcalibration data, we have a 3×3 matrix of chromaticities, a column ofx,y,z for Red, a column for Green and a column for Blue. The inverse ofthe foregoing matrix is dotted (a matrix inner product is formed) withthe vector of white chromaticity values to yield a vector of weights.When the first weight is multiplied by each entry in the first column ofthe original matrix, the second weight by each entry in the secondcolumn, and so on, an RGB to XYZ matrix results. In particular, thesecond row of said matrix holds the Y TSV, or luminance value, thatshould prevail in each channel at the aim white point. The lumeter canbe calibrated as described earlier, in connection with FIG. 30, tomeasure the luminance values so that the system software can decide whatchanges of gain or bias are needed. The foregoing description is meantonly to clarify how the lumeter is used, not to restrict the scope ofthe invention to this model of color mixture.

Next, the gamma is measured in each channel 1205 by commanding a seriesof digital levels and measuring the result. If these need to have veryspecific values, but do not, then it is necessary to re-adjust gain andbias and re-iterate over neutral balance. If the gammas are critical,then the gammas should be adjusted to be close to the target valuebefore neutral balancing. If the gammas merely need to be within a givenrange, then the TRCs in the profile can be adjusted to reflect the realstate of the device once the values are within the desired range. Oncompletion, the profile is updated 1206 and the Lumeter can return tosedentary (standby) mode 1201.

The adjustment of the TRC is accomplished as follows, referring to FIG.30 If the aim value of output for a given command (this is the addressin the TRC lookup table or the independent variable) is 1210, then entertables of aim and measured values as a function of digital command levelto find that level which produces the aim value 1211 among the measureddata. That level becomes the entry or dependent value in the TRC LUT. Inthe interests of avoiding quantization artifacts, it is preferable touse precision greater than that afforded by 8-bit-in/8-bit-out LUTs.

For most video display technologies, with the possible exception ofthose which rely on microfilters to provide wavelength selection (e.g.,LCDs,) the problem of color differences in different regions of anominally uniform color field can be separated from the aspects ofdevice modeling that involve color mixture and gamma. For example,methods for flattening the screen are described in U.S. Pat. No.5,510,851. The sensor and cowel assembly of FIGS. 23 and 24 can providean automatic means of measuring non-uniformity of the display. Aninexpensive, black & white, solid state camera, fitted with a wide fieldlens, may be located and centered in a door assembly in cowel 401 whichcan cover the opening 411 in the cowel 401 (FIG. 24B.) It is preferableto eliminate the effects of environmental light on spatial homogeneitymeasurements. The camera should view slightly defocused full-fieldscreen in the interest of anti-aliasing, one color channel at a time.The digitized images are then analyzed for non-uniformities andcorrections computed and applied, such as described in the above citedpatent U.S. Pat. No. 5,510,851, or preferably as described below inconnection with FIGS. 40-42, and 43A-43B. The distortion function of thewide-field lens should be measured and factored into the calculation ofcorrection factors, as should differential sensitivity across the sensorarray.

The following will describe color measure instruments utilizingspectrographs compatible with high resolution measurement of phosphoremissions from color monitors, i.e., CRTs to provide non-contactmeasurement, self-calibration, and push-button operation. The termpush-button operation refers to the capability of a user automaticallyinitiating color calibration, or that such color calibration isinitiated automatically by a computer coupled to the monitor. As statedearlier in connection with FIG. 27, the sensor of the lumeter can beseparate from the control and interface electronics. Sensor of thelumeter could also be a fiber optic pickup, with collecting optics, thatcould be coupled to a spectrograph. Therefore, the arrangement forspectral monitor calibration is non-contact and automatic once thesensor is fitted in the circumferential cowel.

Self-calibration of a spectral instrument involves insuring thatreadings are associated with the correct wavelengths and have the properrelative or absolute amplitudes. Unless damaged electrically or byradiation, solid state sensors tend to be very stable over time, more sothan most lamps. For this reason, reference detectors are conventionallyused against which standard lamps can be calibrated. Temporal variationsin a source are best dealt with, in reflection or transmissionmeasurements, by a dual beam technique, wherein the light source isreflected (or transmitted) off known and unknown objects eithersimultaneously or successively, depending on the temporal stability ofthe source. The spectral properties of the unknown object are inferredfrom the ratio of its spectrum with that of the known object.

A pulsed xenon source has many advantages for reflection andtransmission spectroscopy. It emits strongly in the short visible waveswhere silicon detectors are less sensitive. It has a very stereotypicalarray of impulsive spectral lines across the visible spectrum which arevery useful for wavelength calibration of spectral sensors. However,output is not stable from flash to flash making them best suited to dualbeam designs.

Referring to FIG. 33, a block diagram of a color measurement instrumentis shown utilizing a spectrograph. Source 1301 shows a pulsed xenon lamptethered over a short distance to its power supply 1302. Source 1310 maybe Xenon modules, such as manufactured by EG&G of Salem, Mass. Two fiberoptic taps are taken, tap 1303 to the reference input of the dual beamspectroscope 1307 and tap 1304 to illuminate the reflection sample. Tap1303 is bifurcated so that light is also taken to an assembly 1305consisting of at least two Light-to-Frequency Converters (LTFCs) whosepurpose will be described presently. Pickup 1306 is a fiber optic pickupwhich can accept light reflected or transmitted from a sample.Alternatively, it can be placed in the sensor assembly shown in FIG. 27,for use in spectral monitor calibration. Although the fiber optics areemployed for flexibility, they must not be flexed during use and thecomponents of the assembly that are linked by fibers must bear a fixedrelationship to one another. The term fiber optic tap refers to one ormore fiber optic cables.

The LTFCs in assembly 1305 serve two purposes. By monitoring theiroutput in the dark, the temperature of the assembly can be estimated aswe have seen earlier. However, they are also used as referencedetectors. Each receives light through a different, spectrally-selectivefilter. For example, two LTFC's may be used, one tuned to a particularpeak in the short wave region of xenon output and the other tuned to apeak in a longer wave region. In this way, they serve as a check on theconsistency of the sensitivity of the reference sensor in thespectrograph at a given temperature. They also aid in estimating thedark current in the reference line camera. In summary, they contributeto checking amplitude calibration of the spectrograph, which may becontrolled by the control electronics of the spectrograph. As notedearlier, the pulsed xenon spectrum provides very predictably placedpeaks for use in wavelength autocalibration. The two aspects ofcalibration discussed here need not be done simultaneously in the caseof monitor measurements; it is enough that they have been done recently.Thus, the signals from the LTFC sensors can be used to provideinformation for automatically checking the calibration of thespectrograph.

Referring back to FIG. 3A, the color measurement instrument of FIG. 33may also be used for reflection color measurement from a hard copysample, or media. The fiber optic probes 36 disposed over the papersample were given a 45/90 geometry. This geometry works well, especiallywhen the illuminating fiber oriented at 45° has a tip of oval shape suchthat the light spot formed is approximately circular with diameter 3-5mm and uniformly illuminated. However, the foregoing geometry may not besuited to tight quarters. Two other configurations also produce verygood results, even when the illuminating and detecting fiber probes havenearly the same orientation (typically near vertical) with respect tothe copy. One configuration includes a polarizer placed over the tip ofthe illuminator fiber optic and a polarizer that is crossed, withrespect to the first, is placed over the detector fiber optic. Specularpickup is severely attenuated, provided that crossing is almost perfect.Alternatively, a diffuser may be placed over the illuminator fiberoptic; in this case, the end of the fiber should be several millimetersbehind the diffuser. This is easier to setup than the polarizers, butmay produce a larger illuminated spot than is desired in somecircumstances. In each of the above configurations, all measurements arenon-contact. Therefore, they do not disturb the copy sample beingmeasured. The configuration we describe does not require significantbaffling of extraneous light due to the very considerable (in a briefinterval) output of pulsed xenon.

The preferred spectrographic design for simultaneous, dual beam work isone described by J. T. Brownrigg (“Design and Performance of a MiniatureDual Beam Diode-Array Spectrometer,” Spectroscopy, Vol. 10, pp. 39-44,November/December '95) and manufactured by American Holographic(Fitchburg, Mass.) as the MD-5. For applications in which thespectrograph and related modules are embedded in a printer, such as anink jet plotter with a single, mobile head the design discussed in thearticle has satisfactory spectral resolution. Other spectrographsproviding comparable performance may alternatively be used.

However, for applications in which the unit (the color measurementinstrument of FIG. 33) is installed in a pen-plotter-like assembly formeasurements of specialized calibration forms and occasionally removedfor monitor duty, a modified design is needed to achieve adequatespectral resolution. In it, the optical bench is lengthened and agrating with superior dispersion and focusing properties is used and thesystem must be aligned to optimize focus in the long wave region of thespectrum.

The pen-plotter assembly transports the paper or copy automatically whena pressure switch senses its presence. The paper/copy is transportedthrough automatically while the pen scans the paper perpendicularly tothe direction of the advance. The fiber optics described above replacethe pen and scans the copy perpendicularly to the direction of paperadvance. Hence, we have a non-contact measurement by an instrument thatis fully self-calibrating and provides push-button simplicity. Thisconfiguration could easily subserve the color measurements needed insupport of soft-proofing as discussed in connection with FIG. 31 andearlier in this description. If one or more instruments of the kinddescribed were available in a network for virtual proofing, they mightbe shared for hard-copy or, when needed, for monitor calibration.

A less sophisticated calibrator may be used for inexpensive printdevices. For “low end” devices, the strategy of choice is to make stockprofiles available for identified lots of inks or toners, through theCyberchrome Service (FIG. 22) and then to use the calibrator to makesure that the devices conform as much as possible to a setup consistentwith the assumptions used in making the profile. This would entail, at aminimum, ensuring that the Tone Reproduction Curves (TRC) for eachchannel in the printer conformed to specification.

Optionally, it would ensure that the image area of the device is flat,i.e., an uniform image reproduces uniformly across and down the page.This would require techniques analogous to those described in the artfor flattening video displays, and referred to earlier in thisdescription. In essence, at each point on the page, the TRCs areadjusted to match the least commonly achievable TRC and some sort ofinterpolation is employed to feather the corrections applied tocontiguous blocks on the page for which TRC corrections were computed. Auseful instrument for measuring the non-uniformities and initial TRCsfor the different color channels is a monochrome scanner such as thePaperPort, marketed by Visioneer of Palo Alto, Calif.

This is an advance over Bonino's patent (U.S. Pat. No. 5,309,257) in tworespects: a) distribution of colorant-lot-specific profiles and b)flattening the page. Each of these have considerable impact on the colorreproduction of any device, especially less expensive ones.

Referring to FIG. 34, a block diagram of a color measurement instrumentutilizing a concentric spectrograph is shown for providing spectralimaging colorimeter. Such a color measurement instrument is preferablyused for page printers, such as a xerographic press. Concentricspectrographs are described by L. Mertz, Applied Optics, Vol. 16, pp.3122-3124, December 1977, and are manufactured by American Holographic(Fitchburg, Mass.) and Instruments SA (Edison, N.J.) Source 1401 is anillumination source. By virtue of the spectral analysis, theinterpretation of object colors can be independent of the source,opening the possibility of simulating the appearance of a product, for acustomer, under various viewing conditions.

Reflector 1402 is a conventional reflector and pickup 1403 a fiber opticpickup which collects light from the reference reflector for conveyanceto the spectrograph. Light from the reference reflector is shown on thesample object. Light reflected from the sample object is formed on a row(streak) of pixels indicated by fibers 1407. Fibers 1407 can be a row offiber optic bundles, or samples (pixels) of the image formed by a lensinstead (A.) Collection of reference light by 1403 enables a dual beamoperation in which the influence of the source illuminant can beextracted from each pixel. Fiber 1404 is a fiber which sees a black trapand is used to generate reference dark current on the array. Fiber 1405is a fiber (or generally, an input) which transmits light from a sourceused for wavelength calibration, such as source providing light of knownwavelength(s).

Surface 1408 is the reflecting surface of a concave mirror whose centerof curvature is shared with convex holographic grating surface 1409;this is a simplified depiction of a concentric spectrograph (B.) Rays1406 _(ab) and 1406 _(bc) show the flow of image rays to imaged pixelsat the entrance slit of the spectrograph and hence to the sensor array1410, on which one dimension encodes wavelength and the other distancein space along the imaged streak (C.) In other words, the concentricspectrograph outputs spectra spatially related to points along the lineof light provided to the spectrograph. Thus, the wavelengths of eachspectrum related to light received from each of fibers 1403, 1404, and1405 can provide calibration reference information for automaticallycalibrating the spectrograph.

The spatial resolution of a practical implementation is such thatanti-aliasing filters, described earlier, are required. Conventionalcompression algorithms must be applied to the spectral data to make itsufficiently compact for storage. Where data need to be captured at highspatial resolution, a separate line camera with approximate photopicsensitivity should be used to capture the gray scale information of theimage, while the chromatic information is coded at lower resolution bythe concentric spectrograph. The approach lends itself tomulti-resolution encoding of images as practiced in a number ofcommercial imaging architectures such as Kodak's PhotoCD and theFlashPix. When used as a digital camera, the sensor arrays must betranslated across the plane of the subject. When applied to on-pressmeasurement of color across the web, the arrays are stationary and theweb moves. Thus, the transport mechanism 320 (FIG. 22) for a physicalcopy and associated light-collecting optics should constitute a moduledistinct from the module which attaches to video display and from themodule containing sensor(s) and control electronics. Light-collectingmodules may be connected to the control module by fiber optic links.

As a spectral, imaging colorimeter, the capture device will require thespeed and image quality insured by a concentric optical design alongwith anti-aliasing software or optical design and with facilities(hardware and/or software) for compressing and manipulating spectraldata and for hierarchical, multi-resolution image storage compatiblewith conventional image data protocol, such as Flash Pix.

Although the above description relates to the printing and publishingindustry, it is also applicable to other industries, such as textileprinting. Further, in packaging and related industries, more than fourcolorants may be used in circumstances in which no more than fourcolorants overlap within a given region of the page. The system canaccommodate this by applying the herein methods to separate regions ofthe page.

Video projection is described earlier in that a video display 17(FIG. 1) can project an image, and a CMI that monitors the output issituated near the source of projected light, as close as possible tobeing on a line of sight. Referring now to FIG. 35, such a color displayis shown in more detail in the case of a front projection color display1500 (also referred to as a color projector). This display representsone of the rendering devices described above that may be used in virtualproofing with one or more other rendering devices.

Projection color video display 1500 has a housing 1501 havingelectronics 1502 which projects light 1504, via projection optics 1506,onto a surface 1503, such as a screen. A color measuring sensor 1508 isprovided in housing 1501, and a beam splitter or partially-silveredmirror 1510 deflects light 1505 onto the sensor 1508 that is reflectedback from the surface and collected by projection optics 1506. Light1504 which reconstructs an aerial image on surface 1503 is produced byelectro-optics of electronics 1502 from image data. As such electronics1502 may be typical of digital electronics, such as amicroprocessor-based system, for optically producing such images, andprojection optics 1506 (illustrated as a lens; may be one or more lensesproviding focus and/or zoom control) for mediating the imagereconstruction which is typical of projection optics used in front colorprojector devices. For example, transduction of digital image data totime- and space-varying signals for projection may be implemented bydigital light processing by digital micromirror devices, such asavailable from Texas Instruments, Inc. However, any other projectionrendering technology may also be used.

Some of the projected light deflected by beam splitter 1510 falls into alight trap 1513, such as a light absorbing block, and is wasted.Reflected light 1505 from surface 1503 follows a path through optics1506 and is deflected by beam splitter 1510 onto sensor 1508. Sensor1508 may be either of an imaging type or a spatially-integrating typeand either color or monochrome. “Monochrome” has a single-rather thanmulti-channel sensor in terms of the number of wavebands of lightdistinctly apprehended by the sensor. Thus, optics 1506 employed forvariable focus in the projection system of projector 1500 is alsoutilized to collect light for the sensor 1508 integrated in housing 1501of the projector. Additional optical elements comprising lenses,filters, etc., may be positioned in path 1510 to 1508 as needed.

The image data provided to electronics 1502 may be provided digitally,through an interface 1512 from a computer (e.g., image data prepared bya presentations software package) or from a network. However, dataderived from analog television signals, for example, are not excluded.Data may also be high definition digital video, or of other typicaldigital video formats. The types of networks providing data includebroadcast television, Internet, cable or any known information-carryingmedium with adequate bandwidth. They may also include Virtual Proofingnetworks. The image data may be “read-out” for projection by a digitalmicromirror device, or reproduction by any technology subserving similarfunctionality, such as Cathode Ray Tube technology or self-luminescentflat panel technologies employing liquid crystals, plasma or OLED(Organic Light Emitting Diodes) whether used for projection orotherwise. Flexible displays, “e-paper” and prints (hard copy preparedby printers or presses) are also relevant rendering devices/media.Alternatively, or in addition to, interface 1512, an optical drive maybe provided, such as DVD, CDROM, or other optical storage device, whichmay be directly accessible to the electronics 1502 of the projector1500.

Optionally, another sensor, or fiber optic pickup routed to such asensor, may be placed at the location occupied by light trap 1513 enlieu of such light trap. The purpose of the sensor would be to monitorthe overall level and/or spectral composition of the projector'sluminous source. Sensors providing this functionality include the CMI'sof FIGS. 2, 3A, and 3B described earlier as colorimeters, such asdiscrete (unitary) colorimeters (SOM 13) or imaging colorimeters(imagical 14) which may be used in conjunction with single-channellight-detectors. The sensor may also have one of optical filters, fiberoptics, or wavelength separating element (e.g., grating, prism, ormonochromator) through which reflected light is received from thescreen. Also, the spectrograph of FIG. 33 may be used, or the sensorsdescribed in U.S. application Ser. No. 09/832,553. As such, the sensormay be a spectral sensor having wavelength resolution adequate toachieve Nyquist sampling of the luminous source, two or more lightdetectors tuned to different, narrow regions of the spectrum or two ormore light detectors with broad, overlapping, non-identical spectralsensitivities. The choice of sensor and strategy for monitoring theprojector's luminous source depend on the optical and physicalproperties of the source and design trade-offs having to do with cost,required accuracy, “footprint” or space considerations within theprojector housing. Under suitable circumstances, the sensor at 1508 maydouble as the source monitor by receiving light routed through fiberoptics from the location of light trap 1513.

For example, if the projector 1500 employs Texas Instruments' DigitalLight Processing (Micro-Mirror Device) technology in conjunction with ametal halide light source, it may be advantageous to monitor the lamp'soutput before it is filtered into Red, Green and Blue channels forcarrying the information of the corresponding channels of digital colorimage data. In this case, it may not be desirable to position the sensoror fiber optic pickup at the location of light trap 1513, where rapidlyvarying filtered light would likely be present. Monitoring the lampdirectly may provide opportunities for control instead of, or inaddition to, those available by monitoring at location of the sensor1508, where light reflected from the display surface is sensed.

Luminous source monitoring is also advantageous for non-projectiondisplays, and as such, the above-described considerations apply to lightsources used in LCD and other flat-panel display technologies.Variations in the source can occasion changes in the system's whitepoint and/or overall luminous output, which can be compensated for byrevising color transformations employed by the digital displaycontroller as described herein and in the patents incorporated byreference herein. Where continual calibration/monitoring is needed ordesired, a sensor 1508 should be selected with sufficiently rapidresponse time that single frames of calibration images can beinterspersed by the display controller and synchronously captured by thesensor.

Another embodiment of the color projector 1500 is shown in FIG. 36. Thisembodiment is the same as that shown in FIG. 35, but without both beamsplitter 1510 and the use of optics 1506 to collect reflected light. Inthis embodiment reflected light 1505 a from screen 1503 returns throughan aperture 1514 in housing 1501 to sensor 1508 via optics 1515. Optics1515 is illustrated as a lens, and may be a single lens or a lens systemenabling variable focus (with or without zoom) for imaging light 1505 aonto sensor 1508. Aperture 1514 may be open or have a transparent window(e.g., plate).

It is advantageous, although not necessary, to have the sensor 1508assembly incorporated in the projector 1500 close to the “line of sight”of the projection housing's assembly because it facilitates makingfocusing adjustments in tandem. The projection lens assembly may beadjusted for variable projection distances by an autofocus mechanismbased on technologies routinely employed in cameras. Alternatively, amanual adjustment may be provided wherein the projection assembly isgrasped and twisted (as with camera lenses) or a knob operates on a rackand pinion (as in traditional “carousel” projectors) and the foregoingadjustment is communicated to the sensor assembly through suitablegearing. In either automatic or manual cases, implementation of thecontrols for focusing light onto the sensor benefit by coordination withthe projection focus mechanism, and appropriate settings may becalibrated at the factory. For example, electronics 1502 may control thefocus setting of optics 1506 and 1515, such as by actuators/motors 1506a and 1515 a, respectively, via signal along connections or lines 1506 band 1515 b, respectively, for moving one or more optical elements ofsuch optics. In this manner, when projector focus (or zoom if provided)is changed for optics 1506 by electronics 1502 (or manually by the user)via signals to drive actuator/motor 1506 a, a proportional change ismade to actuator/motors 1515 a, such that when optics 1506 is focused(and aligned via zoom) onto surface 1503 the optics 1515 image of thereflected light from surface (or a portion thereof) onto sensor 1508 isin focus. In another advantageous embodiment, the sensor system employsimaging electronics and the system performs the role of auto-focusingthe projection display, as described later below in connection with FIG.38A.

In addition to projecting images in accordance with image data,electronics 1502 of FIGS. 35 and 36 enables color calibration asdescribed earlier for virtual proofing in which one or more referenceimages are projected on the surface 1503 by the electronics 1502 and aremeasured by sensor 1508. The sensor 1508 provides to the electronics1502 color measurement data representative of reflected light from thesurface. The projector 1500 is interfaced to a computer system at anode, similar to such color displays described earlier, to controlreproduction of displayed images on screen 1503, whereby the electronicsof the projector provides color measurement data from its sensor 1508 asneeded in performing calibration and/or virtual proofing. Alternatively,the computer system may be omitted at a node and the projector 1500operated remotely via pipe 11 a from another node. Optionally, aseparate computer system is not utilized, and electronics 1502 providethe functionality of such computer system to enable color calibrationand virtual proofing, whereby interface 1512 provides a networkcommunication device to other sites along a network.

The projector 1500 may also provide stand-alone calibration byprogramming the microprocessor of electronics 1502 to project images inaccordance with reference images stored in memory of the electronics1502, utilize its sensor 1508 to provide color measurement data indevice independent units, and then generate color error data bycomparing measured data with the color data of the reference images. Acolor transform is created (or modified) in accordance with the colorerror data, such that the image data to be projected is transformed suchthat projected colors onto surface 1503 match the color of the referenceimages. The color transform may be for example, a look-up-table or colorcurve, for the color channels. Projector 1500 of FIGS. 35 and 36 mayrepresent a home, commercial, or digital color theater projector.

Although it is preferred, especially for projection-type displays, it isnot necessary to have the sensor 1508 assembly incorporated in theprojector because the light emitted by the display screen is generallydivergent or diffuse, such that there is not a narrowly defined angle ofcollection. This can be understood intuitively from the fact that usersare able to see what is on the screens with fairly wide latitude withrespect to viewing angle. However, the further off-axis the sensor is,the greater the need for luminous distortion corrections, particularlyif the sensor is an imaging sensor.

Although an arm member may be attached to the top or side of a largearea display of a “home theatre” system in order to support a sensor orlight pickup disposed to have a suitable view of the screen, users mayprefer another arrangement as shown in FIG. 37. A display 1600 having ascreen 1601 is connected to a television receiver and digital controlunit 1602. Display 1600 may be a typical self-luminous display, such asan LCD color display with a large flat panel screen 1601 which may bemounted on a wall, as shown, or located on a base or stand. Wirelesscommunication between the electronics of the display 1600 and thecontrol unit 1602 is provided by their respective wireless interfaces1604 and 1606 having respective antennas 1604 a and 1606 a. Display 1600and/or interface 1604 have electronics enabling communication withcontrol unit 1602. The control unit 1602 also wirelessly (e.g., RF orIR) communicates with relatively low-bandwidth peripherals, such as acalibration sensor (“CMI”) module 1608 having antenna 1608 a, andsignals via interface 1606 to speakers 1610 situated in the room. Eachspeaker 1610 has electronics providing a wireless interface to receivesignals from their antenna 1611 representing audio to be outputted bytheir speakers, and such speakers may be wall mounted as shown, or mayhave self-supporting cabinet, base, or stand. The digital control unit1602 also has other interfaces 1607 providing one or more ports todevices, such as a cable TV input, a “game box,” a “cable” or other typeof modem (a modem may also be integral to unit 1602) a satellite dish, awired or wireless LAN connection to one or more computers (a networkinterface card or equivalent may also be integral to unit 1602) eitherdirectly or through a hub and router arrangement and the like. Amonginterfaces 1607 is at least one for user interaction (joystick, remotecontrol unit, etc.) supported by an operating program. Indeed, controlunit 1602 may be a general purpose or personal computer withuser-interfacing devices, such as keyboard and mouse and other typicalperipherals. The control unit 1602 enables stand-alone calibration ofdisplay 1600, such as described earlier in connection with FIGS. 35 and36, or represents a rendering device for virtual proofing with otherrendering devices. Optionally, the control unit 1602 and display 1600may be without their respective wireless interface and instead beconnected by cable or wires and/or optical fibers in which the controlunit may be situated in proximity to display 1600. Control unit 1602also may be wired to speakers 1610 and also CMI 1608.

It is common to mount one or more of the speakers 1610, such as speakers1610 a and 1610 b, on an opposite wall 1612, directed toward the screen1601 at an angle not too far away from the normal to the screen 1601surface, as shown in FIG. 37. The CMI module 1608 may be incorporatedwith speaker 1610 a. Alternatively, a stand-a-lone CMI module 1608 canbe mounted directly on the wall 1612 or other structure on the wall,such as may be useful when a speaker is not present to properly positionthe CMI module to view an angle at or near normal to screen 1601.

The sensor-to-screen configuration of FIG. 37 may also be used withrear-projection display instead of a self-luminous display 1600. Therear-projection display may be a typical rear projection display havingelectronics programmed for communication with control unit 1602 toenable stand-alone-calibration as described earlier in connection withFIGS. 35 and 36, and virtual proofing described earlier. If therear-projection display, or display 1600, has microprocessor-basedelectronics which can support the operation of control unit 1602, thecontrol unit 1602 may optionally be removed and its functionalityprovided with such electronics.

A block diagram of CMI 1608 is shown in FIG. 37A having an aperture 1614with a tube or cylinder 1615 in which is fixed a lens or lens system1616 for imaging onto a sensor 1618 (such as a photodiode or CCD). Thesensor 1618 is controlled by imaging electronics 1619 which receivescolor measurement data from sensor 1618 which may be outputted tocontrol unit 1602 via RF interface 1620. Imaging electronics mayrepresent a microprocessor-based device with memory which can carry outprograms such as operating in response to commands from control unit1602, or circuitry for enabling data from sensor 1618 to be formattedfor transmission via interface 1620 to the control unit 1602. Areplaceable battery 1622 provides power to the components within thehousing 1624 of the CMI 1608.

In order to insure that the CMI 1608 senses the screen 1601, and notother areas, the CMI module may be equipped with a laser diode 1626which may be activated by push-button 1627 or by programmed command fromcontrol unit 1602 to electronics 1619 via RF interface 1620. The laserdiode 1626 provides an aiming beam through aperture 1629 onto screen1601. An x-y stage 1628 is coupled by a mounting member 1625 whichattached to the outer surface 1630 of speaker 1610 a, such attachmentmay be by screw(s). The x-y stage may be similar to that used in opticalmicroscopes. The CMI module 1608 orientation is adjustable in twodimensions (a “left-right” dimension of adjustment is suggested by inthe x direction 1627 a, and “up-down” dimension is along the y direction1627 b) until the aiming laser beam aligns with a centered spot, annulusor “bull's eye” of an alignment image illuminated on the display screen1601 by control unit 1602 allowing for a small offset due to the factthat the laser pointer is not centered with respect to the optic axis ofthe sensor 1618. This may be performed manually by the user rotating oneor both x and y adjustment screws 1631 of stage 1628 until alignmentwith feature(s) of the alignment image on screen 1601 is achieved. Toadjust the angular position of CMI module 1608 to view a screen, member1625 is mounted (by screws or bolts) to surface 1603 to provide thedesired angular view (line of sight) of barrel 1615 to the screen.Angular adjustment mechanisms may also be provided, such as by couplingmounting member 1625 to a ball rotatable in a socket mounted to surface1603 (or vice versa), or using a hinge to couple mounting member 1625 tosurface 1603 to pivot the CMI module, in which once at the desiredangular position a set screw locks the ball or hinge in place.Optionally, instead of laser diode and aperture 1629 the imagingelectronics may control movements of stage 1628 until it detects thefeatures of the image of the alignment image on sensor 1618.

Although critical focus will not be necessary, the lens system 1616 mayneed a z-axis adjustment for varying the focus, such as provided byactuator or motor 1632 having a gear engaging an outer thread alongcylinder 1616 to move the tube, and its lens system therein, along thez-direction. However actuator 1632 may be removed, and several differentfixed-focus calibration sensor modules may be made available; in thiscase, the installer/maintainer of the home theatre system would choosethe appropriate one of module 1608 having the proper fixed focusdistance, which may be aligned when mounted on speaker 1610 a and testedat the time of installation. Variable focus may be provided in the waysmentioned above, namely adapting autofocus technology commonly used incameras or manual focus of the sort used in cameras or projectors. Inthe case of manual, variable focus, it will be necessary either toprovide a viewfinder, or to mark the barrel of the adjustment cylinder1615 extending outside housing 1624 with distances between the moduleand display screen 1601 (or to screen 1701 of FIG. 37B). A lock downshould be provided to prevent the cylinder 1616 from moving once properfocus is obtained. An optional display 1633 is shown for enablingelectronics to output the captured image from sensor 1618, similarly tothat of a digital camera, to enable a user to determine when focus isachieved. In addition, a sensor 1618 may be an imaging sensor (e.g.,CCD) which may be operated in coordination with the control unit 1602electronics to seek optimum focus and correct alignment as was describedearlier. Display 1633 need not be built-in, but may be available to aninstaller or user through interface to a hand-held device, as discussedlater.

Among the steady improvements in flat panel technology is included areduction in the degree of directionality of the screens. (In someapplications, directionality is desired, but theatre applications arenot among them.) In other words, variations in luminance and/or colorwith the extent of off-axis viewing have diminished. Nevertheless, in asituation in which a speaker/CMI combination is mounted in a positionthat is at a substantial angle from the normal to the screen, it may benecessary to correct calibration readings for a directional effect. Itmay be possible to do this automatically, based on stored correctionfactors, if the calibration sensor of the CMI 1608 is an imaging device,capable (with software) of estimating the angle of regard (see below.)Otherwise, an approximate correction of readings may be used towardthose that would be obtained along a normal based on the angle of regardprogrammed at the time of system installation, and consistent witheither on-site calibration or values derived from factory calibrationfor different angles of regard.

The CMI module 1608 may be battery-powered as described above.Alternatively, power may be provided through a wire connection, possiblyassociated with an assembly for a speaker, or to an AC adapter to apower outlet in the room. Wireless communication by the control unit1602 (FIG. 37) to other components may utilize a communication protocol,such as Bluetooth.

The sensor-to-screen configuration described for FIG. 37 may also beemployed for front-projection display systems. Accordingly, anotherembodiment of front color projector is illustrated in FIG. 37B showing aprojector 1700 projecting images onto the surface of a screen 1701, asindicated by arrow 1703. As in FIG. 37, the sensor 1608 may be mountedto speaker 1610 a, or to wall 1612. The sensor 1608 receives reflectedlight from the screen 1701, as indicated by arrow 1704. The control unit1706 may be the same as control unit 1602, and able to communicate, viaits wireless interface 1708, to projector 1700, via the wirelessinterface 1710 and antenna 1711 of the projector. The controller 1706provides image data to projector 1700, wirelessly, and also enablescalibration and virtual proofing of the projector similar to thatdescribed earlier for projector 1500 of FIGS. 35-36. The controller 1706may have a network interface device for enabling virtual proofing withsites along a network, or the controller may provide stand-alone colorcalibration for the projector 1700. Optionally, the controller 1706 maybe removed, and the operation of the controller provided withinprojector 1700 by the projector's microprocessor-based electronics. Ifdesired, wires (or fiber cables) rather than wireless interfaces andantenna may be used for enabling communication between control unit 1706and other components in FIG. 37 or 37B.

Placement of the CMI 1608 as shown in FIGS. 37 and 37B at a distancefrom a screen may also be used in theaters, in which the CMI is locatedalong the theater wall near the wall opening through which a digitalcolor theater projector projects light on a screen. Such digital theaterprojector may be calibrated for color similar to the front displayprojector 1700. Purveyors of streaming video or movies-on-demand mayreasonably and advantageously require a level of image and colorfidelity at commercial installations where their products are viewed.

The foregoing considerations apply as well to a camera system that maybe employed to measure or verify uniformity of luminance and/or coloracross the rendering surface. In this case, an imaging sensor issubstituted for a “spot” sensor. Focus is more important in this case,although the anti-aliasing effects of some degree of defocus maynonetheless be advantageous. Technologies mentioned in earlierdisclosures may also be used for said purpose.

Since the sensor's “line of sight” may not be along a normal to thescreen surface in FIGS. 35, 36, 37, and 37B, it may be necessary toestimate the angle of regard and to correct for distortions introducedby the lens system and the angle of regard in software, possibly withspecialized hardware assist. Correction for such distortion isdescribed, for example, in bibliographies compiled by Azriel Rosenfeldon image interpretation in the literature on computer vision, seehttp://iris.usc.eduNision-Notes/rosenfeld/contents.html. Furthermore,once the angle of regard has been calculated, as described below, it ispossible to employ directionality corrections automatically.

Referring to FIGS. 38 and 38A, a flow chart is shown of the process foraligning the CMI module 1608 of FIGS. 37, 37A and 37B with the displaysurface and then correcting for spatial luminous distortion. The CMImodule 1608 (FIG. 37A) is initially aligned when the module is firstmounted onto the frame of the speaker 1610 a or mounted on the oppositewall of a room facing the display surface (step 1851). Next, if theimaging sensor 1618 (e.g., CCD) is not used for autocalibration (step1852), then the lens system 1616 is focused such as by manual rotationof cylinder 1615 (FIG. 37A) using distance marks on the cylinder or,optionally, using viewfinder display 1633 (step 1854). If imaging sensor1618 is used for autofocusing (step 1852), then the process shown inFIG. 38A takes place.

In FIG. 38A, an alignment image is rendered (step 1816), and then theelectronics 1619 sets and records the focus setting for the lens system1616 (step 1817). For example, the focus setting may be set byinitializing the actuator 1632 with respect to the cylinder 1615 at aknown origin position, such as by a stop along the outside of thecylinder which stops movement of the actuator when the cylinder is inits full in or out position. The electronics 1619 then provides one ormore drive (analog/digital) step signals which move the actuator 1632 toone of different focal positions, where each step (forward or back)changes the focus by a known distance. When step 1817 is firstperformed, the electronics may set the focus setting to about half ofthe focal range. Next, the image of the display surface 1601 or 1701 iscaptured by sensor 1618 at that focus setting and then analyzed byprogramming in the electronics 1619 for whether the image was in focuson sensor 1618 (step 1818). For example, the pixels may be analyzed foredges of objects to determine when focus is proper. If OK, the lenssystem 1616 is properly focused (step 1820), otherwise the electronics1619 determines the direction of movement along the z-axis for movingcylinder 1615 and moves lens system 1616 by preset distance(s) stored inmemory of electronics 1619 accordingly (step 1821), and repeats step1818. Such z-axis movement is described earlier in connection withactuator 1632 of FIG. 37A. Once focus is OK (or within a predeterminedtolerance) at step 1820, the process continued to step 1856 of FIG. 38.

At step 1856, alignment is checked and refined, if necessary, by use ofone or more of: (i) the optional aiming beam of laser diode 1626 (FIG.37A); (ii) electronics 1619 providing an overlay upon the images ondisplay 1633 of an outline of the feature(s) and the user (or installer)manually adjusting stage 1628 in one or more dimensions to align thefeatures imaged with the overlay (also the angle of view may beadjusted, if an angular adjustment mechanism is present); or (iii)electronics automatically detecting if the captured image has certainpixels or pixel regions aligned with the imaged features and if not,moving stage 1628 in one or more dimensions until such alignment isachieved. Alternatively, the control unit may provide automaticdetection and alignment by sending commands to the electronics of theCMI module to move stage 1628 in one or more dimensions. At step 1857,the control unit 1602 or 1706, operating in a test and setup mode,displays a spatial calibration pattern stored in its memory, such as oneor more square outline figures at high contrast. The CMI module, havingbeen aligned and focused at 1851-1856, captures the test pattern (step1858). The data may be verified to see if they reveal misalignment orlack of focus (i.e., checking if pixels or pixel regions align with oneor more features of the pattern, and measuring pixel width of edges),then a comparison is made of “ideal” image from memory and the capturedimage as to difference in pixel value in one or more color channels(step 1862). Although only one spatial calibration pattern is described,steps 1857-1862 may be performed using a number of spatial calibrationpatterns.

Preferably, the actual luminance profile represented by the “ideal”(possibly for each color channel) of the displayed pattern has beenmeasured as part of a factory calibration, if not for each individualscreen, then at least generically for a group of devices. From thecomparison, the program of the control unit determines at step 1864 thespatial luminance distortions and prepares corrections that will beapplied in subsequent calibrations and procedures. Correction functionsmay be plural in the foregoing if data are collected for each colorchannel of the rendering device. If depth of field is not adequate tofocus the entire test image, focus should be optimized for the center ofthe imager's field. The imager can be re-focused, as necessary, in orderto verify that corrections properly handle the periphery of the displaysurface.

Use of an imaging sensor 1618 for measuring non-uniformity of therendering surface was described earlier by use of an imagical to acquiredata to flatten the screen, i.e., make it regionally homogeneous inlight output or to compensate for interactions among adjacent pixels incomplex imagery, and then later in describing color differences indifferent regions of a nominally uniform color field for a video displayscreen, for measuring non-video display, and computing corrected ToneReproduction Curves (TRC's).

Referring to FIG. 39, an imaging sensor is shown useable withconventional computer (including notebook or portable) video monitorswhich may also be used as a camera for capturing images of the user. Anopened portable, laptop or notebook computer 1900 has a display portion1902 with a screen 1904 and arm member 1906. The arm member 1906 isshown in FIG. 39 pivoted to the open position and at the end of the armmember is a pivot joint 1908 for mounting a camera 1910. The camera maypivot as indicated by arrows 1911 along joint 1908. The joint is shownin more detail in FIGS. 39A and 39B, in which camera 1910 is coupled bya shaft 1912, which may be part of the camera. The end 1913 of the shaftis shaped to be received between two flanges 1914 extending from the endof arm member 1906. An opening 1915 (shown in dashed lines) extendsthrough flanges 1914 and shaft end 1913, and a pin 1916 is receivedthrough opening 1915 to pivotally mount shaft 1912 to arm member 1906.End 1913 is rotatable about pin 1916 in the socket formed betweenflanges 1914. The other end 1917 of arm member 1906 forms a pin that ispivotally mounted in a grooved slot at one end of a recess 1918 in thetop of display portion 1902, such that the arm member can pivot to afull open position (as shown in FIG. 39) or can pivot to a closedposition within recess 1918, which is shaped to received the arm memberand camera 1910 when the camera is not in use. Arrow 1920 indicates thepivoting of arm member 1906 and camera 1910 from open and closedpositions. Other mechanisms for pivotally mounting arm member 1906 toboth computer 1900 and camera 1910 may also be used.

Camera 1910 is preferably a small color CCD-based camera of the sortwhich has been adapted to clip onto the frame of display portion 1902and used for video conferencing. The camera 1910 can send digitalsignals to computer 1900, via a cable, to a port of camera 1910, wherecamera 1910 provides output signals using communication protocol forsuch port. For example the port may be a USB port using USBcommunication protocol, or other port or interface (e.g., card) andconventional communication protocol may be used to enable communicationbetween camera 1910 and computer 1900. Preferably, such connection tothe computer is via a cable 1910 a to camera 1910 extending through oralong the outside of arm member 1906, as shown in FIG. 39, by a dashedline along member 1906.

Camera 1910 is modified (possibly involving an adjustment of the focaldistance for the camera) to attach and pivot about on an arm member 1906from the frame surrounding the display surface. If display portion 1902is too thin to accommodate the camera in recess 1918, then camera 1910can be provided separately to snap into pivot joint 1908 of arm 1906,being held in position by engagement of one or the other detent/buttoncombinations described below. Alternatively, shaft 1912 and a bore inthe camera 1910 may each be threaded to enable joining them.

The camera 1910 may be pivoted to “see” the user, such as in a videoteleconferencing application, and may be pivoted to face the screen 1904and thus “see” the screen in a calibration application. Thus the camerahas “dual use.” Detents are provided on surface 1913 a of shaft end 1913and engage one or more buttons along surface 1914 a of one or both offlanges 1914 as it pivots and thereby temporarily fixes the rotationalposition of the camera 1910 when rotated by the user. Alternatively, thedetents may be provided along surface 1914 a and button(s) along surface1913 a. When the arm member 1906 is in the open position, and the camerais fully rotated to its last detent position to point the camera to thescreen, the camera's optics are automatically aligned with the part ofthe screen to be used for calibration of the color of the screen and/orvirtual proofing with other rendering device(s), as described above.Sense switches may be associated with detent positions capable ofsignaling via a cable or wire 1910 a to computer 1900 which “dual use”mode is currently actionable.

The dual-use camera 1910 may require a bi-focal adjustment with dual,manual detents capable of securing the focus optics of the camera inposition for teleconferencing or calibration applications. As above,corrections for various spatial distortions of the image are appliedbefore measurement non-uniformities are processed for making“field-flattening” compensations. The arm member 1906 need not beembedded in the frame surrounding the screen as shown in FIG. 39, butmay be an attachment.

While an imaging sensor is very useful in detecting spatialnon-uniformities of luminance and/or color across the display surface,it may lack the sensitivity required to distinguish low light levels ina careful calibration of display gamma or tone reproductioncharacteristics. It is advantageous to compensate for reducedsensitivity, at least partially, by forfeiting spatial resolution infavor of increased sensitivity and integrating light responses over manypixels when accurate low-light measurements are important. In theforegoing situation, uniformity correction should be made at relativelyhigh luminance on the display, followed by spatially integratedlow-light measurements.

Instrumental embodiments and methods described herein are adaptable tohand-held and miniaturized display devices, including cellulartelephones, personal digital assistants and the like. Display technologymay produce very high resolution, compact arrays of micropixels capableof conveying all the information of conventionally-sized computerscreens, especially if viewed by the user through a magnifier. Manycellular phones are equipped with cameras, some of which are capable oftransmitting a series of images as a video data stream. Often, speakerphone functionality is included. Many cameras are capable of autofocusand of imaging at relatively close range. However, the cameras are notgenerally positioned in the handset in such a way as to enable imagingof the handset display, and such handsets are used in environments wherethe user has little control over environmental (ambient) conditions,including illumination. In the following, several structural andfunctional improvements are described that increase the range ofapplications of handset cameras.

FIGS. 39C and 39D illustrate a handset device 1930 having a housing 1930a with upper and lower housing portion 1930 b and 1930 c, respectively.Upper housing portion 1930 b is pivotally attached by a joint 1933 toenable rotation to either a closed position or various open positionswith respect to lower housing portion 1930 c. For purposes ofillustration, handset device 1930 is illustrated as a typical cellularphone with a display screen 1931 (e.g., a color LCD screen) and aspeaker 1932 along upper housing portion 1930 b. Along lower housingportion 1930 c is a keypad 1934, a rocker switch 1935 for speaker volumecontrol, one or more microphones 1938, and a joystick-like control 1936.With the handset in an open position, the control 1936 is used fornavigating through menus presented on the display screen 1931 to providea user interface enabling the user to make menu selections byinteracting with options shown on screen display 1931, as typical of acellular phone. Button switches 1937 may be present for use inconjunction with control 1936 in the user interface, as also typical ofa cellular phone.

At least one switch or trigger 1939 along housing 1930 operates a camera1943 which is disposed in an extension 1930 d of lower housing portion1930 c. The camera 1943 has a sensor element 1944, such as a color CCDarray, and a lens system (one or more lenses) 1943 a through which thesensor element captures images. Lens system 1943 a may optionally havemovable actuators to move two or more lenses to provide focus and/orzoom, typical of a portable digital camera. Optionally, a flash may alsobe present in extension 1930 d for adding illumination if needed tocapture images. The upper housing portion 1930 b is pivotable alongjoint 1930 to multiple detented positions 1941, 1945, and 1946 withrespect to lower housing portion 1930 c, as illustrated by arrows 1945a, where the default position (like a closed book) is illustrated by thedashed lines and fully open by dash and dot lines. Camera 1943 may be aminiature, digital camera of the type commonly used in handsets. Thecamera is connected to control electronics 1951 of the handset alonglink 1952, which may be an extension of an internal bus of the controlunit or a variety of high-speed serial bus interface (e.g., Firewire,Universal Serial Bus, etc.). In the former case, memory-mapped i/o,direct memory access, etc., are typical functionalities. However, anyconnectivity of sufficient bandwidth may be used. A connector 1961 aalong housing 1930 enables external charging of a rechargeable battery1961 b for supplying power to control electronics 1951 and othercomponents requiring power for operation, as typical of a cellularphone. The connector 1961 a additionally may provide one or more portsto the control electronics to interface the control electronics toexternal devices or computer systems.

The pivoting mechanism for opening the handset device provides detentsfor position 1945, enabling the camera 1943 to view the display screen1931 at an angle of between 45 and 90 degrees, while at position 1946the upper housing portion 1930 b is open wide enough so as to be out ofthe field of view of the camera 1943. It is advantageous to have thecamera 1943 be pivotable between two positions (illustrated byfield-of-view cones 1947 and 1948) in order to optimize the angle ofview or line of sight to the display screen 1931 in position 1945 whileinsuring that the upper housing portion 1930 b is completely out of viewotherwise. In the case of a non-folding handset, the camera can bedisposed on a miniaturized arm in a manner similar to the configurationof FIG. 39.

The control electronics 1951 of the handset device 1930 is shown in theblock diagram of FIG. 39E. A controller 1952 controls the operation ofthe handset device. Controller 1952 may be a microcontroller,microprocessor, or a conventional, programmable device which can beconnected to memory 1954, at least part of which can be non-volatilememory for storing operating programs, calibration coefficients, time,date, GPS coordinates, or other data needed for operating the handsetdevice. The controller 1951 is shown connected to peripheral modulesalong bus 1953, which may employ wires and/or optical fibers withsuitable signal converters. The bus 1953 carries address and datasignals. Image memory 1955 is general-purpose and may store text data,contact lists, appointment data, digitized audio data, extended programinstructions and data (including operating system and applicationprograms), in addition to image data in a variety of formats (e.g.,JPEG, TIFF, PDF, or other formats). Image memory 1955 may include Flashmemory, and/or removable memory storage, such as a memory stick, compactmemory cards, or other removable storage.

A transmitter/receiver 1956 is a block comprising circuitry (e.g.,signal transducers, A/D and D/A converters, modulator/demodulator ormodem, or other electronics needed to send and receive voice and/ordata) typical for enabling wireless (RF) communication over an antenna1956 a. It may also comprise a Network Interface Adapter enablingEthernet communications and Voice-over-Internet-Protocol (“VoIP”) whenthe control software is set for such a mode of operation. Serial BusInterface 1961 is a block comprising circuitry for communicating withexternal devices, via one or more ports of connector 1961 b, by serialcommunication protocols, such as one or more of Bluetooth, Firewire,Universal Serial Bus (wired or not), infrared signaling, or standardssuch as RS232 and the like. However, other communication means may alsobe used. When present in the handset device, the serial bus interface1961 may provide alternative access to intelligent print devices (forexample, EXIF-capable printers) and/or to the Internet (for VoIP and thelike) by way of other computers. Camera 1943 in addition to sensor array1944 has signal-conditioning circuitry suitable for converting raw imagedata captured by the sensor array into digitally useable form,preferably JPEG- or TIFF-encoded images with EXIF-compatible headerfields or an equivalent. Camera 1943 has at least the same or betterresolution (both spatial resolution and contrast or dynamic range) ascameras currently available in handset devices. Handset device shown isnot limited to cellular phones, but may be any handheld device having acamera and communication capability, such as digital cameras withwireless communication means for data and/or voice, and personal digitalassistants having a camera and such communication means.

The circuitry associated with camera 1943 may include a frame store orbuffer. Alternatively, a single frame buffer 1959 may be shared bycamera 1943 and display 1931. Signal conditioning for display has beendiscussed elsewhere throughout this disclosure and in U.S. patentapplication Ser. No. 09/832,553, filed Apr. 11, 2001.

Audio signal conditioning module 1958 includes microphones 1938 fortransducing sound waves into electrical signals and one or more speakers1932 for converting electrical signals into sound waves.Signal-processing software and/or hardware may be provided that enablesvoice and speech recognition so that a user may set the handset deviceinto a mode in which menu choices can be made verbally rather than bytraversing a sequence of keystrokes on the handset. Alternatively,controller 1952 may have a network interface enabling it to offload thetasks of speech recognition and command parsing to a suitably programmedcomputer of greater capability. Although control electronics 1951 isillustrated with a traditional bus architecture, other internalcommunication architecture may be used. For example, peripheral modules1931, 1943, 1955, 1956, 1958, and 1961 of FIG. 39C may communicate withthe controller 1952 using a serial protocol similar to that standardizedin the Universal Serial Bus interface (such as described atwww.usb.org).

FIG. 39F illustrates a hierarchical menu structure, such structure beingsimilar to that typically used in portable devices, such as digitalcameras and cellular phones. The figure shows several, nested levels ofchoices in order to reveal the relationships between the levels.However, only one level of selections need be available on the displayscreen 1931 at a given time. The arrow at the top left indicates thatCamera Functions 1966 is a submenu to a higher level in the hierarchy.For illustrative simplicity, four options/selections would appear on thedisplay under Camera Functions: Take Pictures, Video Conference, AutoBacklight/Display Contrast Control, Calibration, and Exit (or Back) to aprevious screen.

Take Pictures 1967 selection leads to another level of selections toinclude configuring EXIF header information (or an equivalent) andcorresponding image capture settings, e.g., flash setting (if a flash ispresent), electronic shutter speed, resolution, or other settingstypical of digital camera or cellular phone having a digital camera.

Video Conference 1968 enables the user to set up the handset device fordisplaying data during a telephone conversation. Such data could includeimages of one or both conversants (if both, a split-screen display wouldbe optional on setup) or of other data such as a textual selection froma document or numerical data from a spreadsheet, or the like. Submenuoptions may be used to select the type of display, entry of addresses orphone numbers of the other party(s) device to be conferenced, orparameters for configuring the video conference.

Auto Backlight/Display Contrast Control 1969 presents a toggle, either“ON” or “OFF”, which can be switched to the other state, for example,when the user employs controls 1936 and/or 1937 (FIG. 39C). When in the“ON” state, Auto Backlight operates, as elaborated below, according toeither threshold 1971, graded 1972 or Boolean 1973 modes selectable byuser-interface controls on the handset device. In an advantageousembodiment, the audio-signal-conditioning circuitry 1958 (FIG. 39C) ofthe handheld device may be used in the conversion of user manual datastored in memory of the device into computer generated speech which canbe activated by the user to provide audible help on demand.

Selection of Calibration 1974 invokes a submenu that enables the user tochoose to calibrate, for example, either the display of the instanthandheld device 1975 (e.g., display screen 1931) or another display (orother color rendering devices) 1976. The options available forcalibration will be explained further below.

In a first improvement in handset functionality, the user interfaceassociated with the display 1931 enables a setting whereby the camera1943, however positioned on the handset, monitors the ambientillumination whenever the user opens the handset device or otherwiseindicates preparation for use. Currently, many handset devices turn on abacklight of a liquid crystal display when preparation for use isindicated whether or not the ambient illumination conditions providecontrast sufficient for reading what is displayed. This may reducebattery life unnecessarily. Then, typically, the backlight is turned offafter a fixed time interval whether or not the user needs enhancedcontrast. Through the interface illustrated in FIG. 39F the user isenabled to choose among, for example, three modes of conserving batterylife consistent with improved display legibility when Auto BacklightControl is set to ON, as shown in more detail in the flow diagram ofFIG. 39G.

The flow diagram of FIG. 39G should not be viewed as one that would betranslated directly into computer program source code. Rather, it isintended to illustrate the Auto Backlight functionality in terms of afew, possible implementations and in relation to handset modules, bothhardware and software. Similar to FIG. 39F, flow begins at the upperleft block 1981 where the user has selected Camera Functions 1966 withinthe menu hierarchy, and then selects at step 1982 a menu option to turnbacklight control ON or OFF via keypad 1934 or controls 1936 or 1937.Optionally, controller 1952 reads at step 1982 the user input bydeciphering verbal commands processed through the audio module 1958. ForAuto Backlight control, the controller 1952 at step 1983 displayssubmenu options and receives the user selection of either a threshold,graded, or Boolean setting, initiates data collection by the camera 1943(i.e., camera automatically captures one or more images to provide suchdata), and then analyzes the data at step 1984. One collection/analysismethod is to average the light distribution for the values of pixels inone or more color channels in images across the camera's sensor array intime and space to produce a value. However, other methods may be usedthat analyze for spatial or temporal patterns in the light distribution.

A check is made to determine if the user selected a threshold setting(step 1985). If so, then the controller 1952 compares the result ofanalyzed camera data to a threshold value stored in memory 1954. If theambient illumination (reflected in the results of analysis) were lessthan a stored value based on prior study of display legibility (step1985 a), then the controller would direct the electronics of display1931 to turn on the backlight (step 1985 b), otherwise the backlight isoff. The controller 1952 may periodically read camera data (step 1986)then repeat steps 1984, 1985, 1985 a, and 1985 b to insure that ambientconditions continue to warrant illumination of the display. Readings maybe taken so long as the threshold condition is met, the handset devicecontinues to be open, or until the camera is put into an incompatiblemode by an external event such as user choice.

If threshold setting was not selected at step 1985, a check is made todetermine if user selected a graded setting (step 1987). If so, a storedfunction describing luminous intensity of the backlight needed forlegibility as a function of the level of ambient illumination is used tomodulate the level of the backlight (step 1987 a). For example, thelevel of backlight could be set to the reciprocal of the average (overspace and time as discussed earlier) ambient light reading where saidreading is clamped at minimum and maximum values found in prior study torepresent levels at which maximum or minimum, respectively, backlight isrequired for readability. This clamped function is illustrated, forexample, in the graph of FIG. 39H. Battery life would be enhancedconsistent with display legibility if no more backlight than necessarywere utilized.

If neither threshold or graded setting were selected at steps 1985 and1987, then the Boolean setting has been selected, and at step 1988 thecontroller 1952 turns on the backlight only if an ambient thresholdcondition (similar to step 1985 a) were met AND the time since the lastkeystroke via keypad (or depressing of controls 1936 and 1937) were lessthan a programmed value stored in memory 1954 (e.g., 15 seconds). At theexpiry of the programmed interval, the handset would depart AutoBacklight mode. Boolean may not be well-suited to the situation in whichthe conversant needed to view variable data, such as text or images,being received for display during a conversation. Threshold, graded, andBoolean are exemplary implementations not meant to be limiting, otherfunctions or conditions may also be used in accordance with the amountof ambient light measured by the handset camera.

Referring back to FIG. 39F, selection of Calibration 1974 of the instanthandset 1975 enables the user to select one or more of three options:Spatial Uniformity, Color and Tone, or Reflected Ambient. SpatialUniformity involves using the camera to measure non-uniformities acrossthe display surface, preferably absent extrinsic or ambient sources ofnon-uniformity. Resulting data are used to homogenize display brightnessregionally or spatially, preferably while maintaining a specifiedneutral balance. Spatial uniformity detection and correction of adisplay screen is described below in connection with FIGS. 40-43B. Colorand Tone involves calibration for one or more of white point (neutralbalance), linearization and color mixture modeling, as describedearlier. Reflected Ambient refers to the estimation of the amount oflight more or less diffusely reflected off the display surface of aself-luminous or emissive device.

In general, cameras of handheld devices, digital cameras, or handsets,are non-linear, colorimetrically. In other words, linear models of colormixture are not strictly applicable in the development oftransformations for converting device codes into device independentcolor coordinates which may subsequently be converted intodevice-specific display codes. Digital cameras generally belong in thecell at the bottom left of FIG. 4C. Methods for preparing approximatecalibration transformations for such devices are described both earlierand also later below with tools (e.g., calibration targets, etc.) forenabling a user to perform such calibration with the theatre system oron the aftermarket. With knowledge of the spectral sensitivity of thecamera's color channels and the likely properties of sets of colorantsand/or illuminants to be measured and characterized, a more exact,colorimetrically, calibrated transformation may be provided, asdiscussed below. Earlier, methods/apparatus were described fordetermining the spectral sensitivity of a sensor; cameras may becalibrated at the factory and calibration data (including“normalization” functions that correct for non-uniform sensitivityacross the surface of an area sensor) may be stored for possibleinclusion in headers to image data files. Such headers are provided forby the EXIF standard, for example. Alternatively, camera calibration maybe provided as an aftermarket service.

A system similar to the Profile Server 316 discussed earlier inconnection with FIG. 22 may be provided to assist in using the handsetcamera as a calibrator. This system, referred to hereinafter as theCalibrator Camera Server, may represent a computer server system at anetwork addressable site, such as via the Internet or World Wide Web.The Calibration Camera Server has memory storing a database of thesensor spectral sensitivities for different sensors, and video displaycolor properties needed for accurate calibration for different colordisplays or other color rendering devices. Such Calibration CameraServer may perform the underlying computations of color transformations,as will be discussed further below. The accuracy of the colortransformations desired dictates the accuracy and the methods requiredin preparing them. Either the handset (via a wireless networkconnection), or the color device to be calibrated (e.g., via a NIC,modem, or other network interfacing means) may be provided with networkaccess for enabling connection to the Calibration Camera Server to queryfor desired information from the database. The Calibration Camera Servermay also be part of the Profile Server 316.

The use of a handset to calibrate another display 1976 has applicationto the calibration of digital home theatre. In connection with FIGS. 35through 38A, several embodiments for calibration, including buildingsensors into projectors and associating them with wall-mounted speakers,were described. However, handsets provide viewfinders (by usingreal-time video images captured by the camera on handheld device'sdisplay) and, potentially, the handset's speaker(s) may convertdocumentary calibration procedures into audible instructions for a homeuser to facilitate calibration with the handset (e.g., for the user whodoes not care to read manuals or develop expertise in calibrationprocedures).

On selection of Another Display 1976 (FIG. 39F), the controller 1952(FIG. 39E) of the handset seeks to establish communication with thecontrol unit 1602 or 1706 (FIG. 37 or 37B) of the other display throughinterfaces provided by transceiver/receiver 1956 or serial interface1961 (via cable or wireless). Unlike FIG. 37 or 37B, no CMI module 1608may be present, but the user (or a service technician) wishes tocalibrate the color of the display using a handset (which as statedearlier refers generally to a portable device having a digital camera,which preferably can be carried in one's hand). A calibration programwould already have been initiated in the control unit for the otherdisplay which prompted the user to select a camera as the measurementinstrument. The user would then be led through the following procedureby prompts that are either displayed on a part of the theatre displayscreen or “spoken” to the user by the handset speaker(s) and/or speakersof the theater system:

Step 1: Control unit 1602 or 1706 of the theatre display systemacknowledges establishment of communication with handset; if a cellphone, the handset appears “busy” to incoming calls. In the case wherethe controller 1952 and/or memory 1955 of the handset is limited, theinitial inter-device identification and “handshake” may involve atransfer of programs and reference calibration data from the theatresystem to controller 1952. Alternatively, the control unit 1602 or 1706may “take over” and treat the handheld device as a multifunctionperipheral. The communication may be according to a USB or USB-likeprotocol, which latter has been developed to support both wireless andmultifunction device interfaces (such as described at www.usb.org).

Step 2: An alignment target is displayed on the theatre display. Imagedata for such alignment target (e.g., cross-hair(s), geometric shape(s),outlines, or the like) are stored in memory of control unit 1602 or1706. The user is instructed to operate the handset's camera so as tocapture an image of the displayed alignment target. Alternatively, onreceipt of instruction from control unit 1602 or 1706, the camera entersa “free run” mode in which video images are captured regularly (e.g.real-time) and displayed on the handset display.

The user is guided in this by using the handset display as a viewfinder.A magnifier may be provided with the handset, and will be especiallyuseful if the handset has a high-resolution, miniaturized display, asmentioned earlier. An example of such magnifier is shown in FIGS. 39Mand 39N. The magnifier represents a lens 1931 a which may be attachablefor adjustment over a display of handset 1930 when such display is aminiature high-resolution display 1931 b. The lens 1931 a is preferablyattached by a pair of arms 1931 c, each pivotal along joints 1931 d(e.g., about pins through arms and fixed members 1931 e) to enableextension and retraction by the user, as needed. Each one of the arms1931 c is attached to one of the sides of lens 1931 a and along display1931 b. Magnifier 1931 a is advantageously removable such that display1931 b may be used for conventional resolution, as by clustering oraveraging groups of adjacent pixels to reduce effective resolution.Alignment will be facilitated further if the proper centering of thealignment target is signaled by a comparison of features of the capturedimage with digital indicia of a reference image stored or loadable intohandset memory 1955 from control unit 1602 or 1706. Image data may betransferred from the handset to control unit 1602 or 1706 andcomputations carried out there to determine when the camera array isaligned with the theatre display and then to notify by voice or visualmessage on the handset and/or theatre display when alignment iscomplete. As it may be difficult for the user to later maintain suchalignment, the handset may be stabilized on a surface, such as on atable, stand, or tripod, or on a support described later in FIG. 39K.

Step 3: Once the handset's camera is properly aligned and the user hasselected the kinds of calibrations 1976 (FIG. 39F) to be performed, thecontroller 1952 and the display unit's controller 1602 or 1706 interactto perform the calibration procedures described herein. In very briefreview, this involves displaying calibration forms or images on thetheatre display, capturing images of the images with the camera,analyzing said captured images and computing appropriate transformationsso that colors of the digital theatre data are displayed correctly.

In order to simplify more detailed discussion of how the calibration ofanother display proceeds with the aid of a handset (or other handhelddevice, such as a digital camera), assume that a calibration program hasalready been initiated through user interaction with a graphical userinterface (GUI) operating through controller 1602 or 1706. Assume, also,that at least one of the channels of the handset camera (e.g., the greensensor channel) has been calibrated such that differences in responsesamong the population of sensor elements to a flat field are normalized,and that the output of the sensors is nearly linear with the logarithm,for example, of the light intensity on the sensors. Functions other thanthe logarithm may be used; what is most important is that the functionbe known. Such calibration is performed as part of handset manufactureor through aftermarket services.

It was shown in Table Two and attendant discussion that the “lumeter”whose spectral response function approximates the human photopicluminosity function tracks a true luminance meter reasonably closely inthe measurement of a gamma function. The closeness of correspondence ofmeasurements taken through the green channel of a handset camera withthose of a true Luminance meter (having the human photopic luminosityfunction) may depend on how closely the green channel responsitivityapproximates CIE Standard Observer's y-bar Color Matching Function. Forthe common scenario of “good enough” calibration, the correspondence islikely to be adequate for most handsets. In other words, performing agamma calibration, such as that illustrated in Table Two, with thehandset is likely to be close enough. Scenarios in which more accuratecalibration of the handset is required or desired will be discussedlater.

Returning, briefly, to user selection of Another Display 1976 underCalibration of FIG. 39F, it is preferred if the choices are accompaniedon the displayed menu by check boxes (or other graphical selectioneffect) enabling the user to indicate all at once which functions are tobe performed. Once selected, the calibration program stored in thehandset implements the functions once the user has set up and alignedthe handset camera for proper viewing of the display surface to becalibrated. It may be advantageous to perform basic Color and Tonecalibrations prior to Spatial Uniformity, even if the user has requestedonly the latter. Likewise, measurement of Reflected Ambient, the amountof light reflected from the display surface when it would otherwise becompletely dark, should precede any other calibrations. As has beenmentioned previously, “dark light”, “background” etc. should be factoredinto other calibrations and/or the user should be advised thatprevailing ambient will compromise calibration and the accuracy of colorreproduction.

Turning now to FIG. 39I, a flowchart shows in more detail the operationof the handset in response to selection of Another Display 1976 by theuser. Although the following refers to calibration of Another Display asbeing one associated with a theater display system, such calibration maybe of other color displays or surfaces upon which rendered color imagesare displayed which may or may not be associated with a theater displaysystem. Further, such calibration using a handset camera is not limitedto displays, but may be of other types of rendering devices which outputcolor images onto media, such as printers, copier, presses or the like.Such media can be fixed in a position for capture by the handset camerato enable alignment with outputted alignment target(s) and then imagesof other outputted media can be similarly captured for calibrationprocessing. First, the user has initiated a calibration exercise on thehome theater system, for example, as an option in a DVD-viewing programor the like (step 1876). The program operating on the theater systemchecks whether there is a calibration peripheral of one of the kindsdescribed earlier (e.g., sensor 1508 or CMI module 1608) associated withthe system (step 1877). If not, the program prompts the user toconfigure the handset camera, through a menu such as that illustrated inFIG. 39F, as a peripheral for calibration of another color display (step1878). If communication between handset and display system is notestablished successfully (step 1879), then the calibration program isaborted with an error message to the user, sent via handset or thedisplay system (step 1882).

If handshaking occurs successfully at step 1879, then alignment andfocusing are verified for calibration, such as shown in FIGS. 38 and 38A(step 1880). In the case of a handset camera, the user is guided throughset up of the handset for proper alignment and focus using the handsetdisplay as a viewfinder, preferably with the aid of a support base (asshown in FIG. 39K and discussed later below).

Regardless of which calibration options were selected by the user atAnother Display 1976 (FIG. 39F), it is advantageous and preferred toestimate the amount of reflected ambient prior to performing othercalibration functions (step 1881). Accordingly, the theater display isdriven with digital codes of 0.0 (black) over its array of pixels ineach color channel. The handset camera captures an image of the displaywhen so driven to black. The captured image is then received andprocessed by controller 1602 or 1706; the image represents the sum ofintrinsic dark light and reflected ambient as a function of spatialposition. Alternatively, such processing could be carried out by thehandset. It is preferred that the handset camera be configured andpositioned such that specularly reflected light is not collected or, atleast is minimized.

Light so measured is compared with manufacturer recommended tolerancesor historical, normative data, such as may be stored in memory of thecontrol unit 1602 or 1706 (step 1883). If ambient conditions are suchthat the theatre display cannot be operated in a calibrated mode due tohigh level of measured dark light at step 1881 or high level ofspatially non-uniform dark light between highest and lowest measuredvalues (step 1883), the user is advised that corrective action should betaken, if possible (steps 1884). For example, the user may turn off alamp in the room or shade windows, and then repeat steps 1881 and 1883.With suitable user acknowledgement 1885 (and amelioration), a renewedcycle of ambient estimation (step 1881) is undertaken. At this point, itis more practical if the user interacts with the system via menusprovided by the theatre display calibration program, rather than throughthe handset, that is, the handset functions only as a color measurementperipheral under the control of the display calibration program oncehandshaking of step 1879 has occurred. If there is no suitable useracknowledgement at step 1885, or several ameliorative attempts arefruitless, the calibration program can either abort at step 1886(releasing the handset) or continue with a less-than-optimalcalibration, depending on the severity of the ambient backgroundproblem.

If the measured reflected ambient is within tolerance at step 1883, theambient measurement data is stored in memory of the control unit 1602 or1706 (step 1887) for use in further calibrations, communication to othersites, etc. If other calibration options (Color & Tone and/or SpatialUniformity) were selected (step 1888), then aim values for Color andTonal calibration are loaded (step 1889), else the calibration programexits, releasing the handset. The aim values and profiles are stored inmemory, and/or downloaded to such memory of the theater display systemfrom the Calibrator Camera Server, which stores in its database suchcolor aim values. As stated earlier, the theater display system and/orthe handset may have accessibility to the Calibrator Camera Server toquery and receive the requested aim values. Optionally, the user mayspecify such color aim values via a menu of a graphical user interfaceprovided on the theater display or default values may be used. In morecomplex scenarios, requiring more accurate color calibration, colorprofiles may be computed from data collected at the site; this will bedescribed briefly below in connection with FIG. 39J.

The aim values of step 1889 represent the color target values, such asat least one set of gamma curves and a white point that are associatedwith a color profile capable of converting color values intodevice-specific values for the theater display. Typically, this profileconverts from one set of RGB coordinates to another by amulti-dimensional transformation. The profile may be formatted accordingto the ICC standard and/or be compatible with Virtual Proofing.

At step 1890, the handset camera measures the gamma curves of thetheater display color channels. For example, successive screens may bedriven to output color in accordance with programmed pixel values, andsaid screens are captured as images by the handset camera. Each capturedimage is received and stored in memory of the theatre display system,and then processed by controller 1602 or 1706 to determine the currentgamma curves of the theater display. Also, the display screen is drivento the color white (i.e., each pixel in each color channel set to itshighest value), and then captured as an image by the handset camera. Theimage is processed (e.g. averaged over all or a subset of pixels) toobtain the measured white point. The measured set of gamma curves andwhite point are then compared with the aim values for the gamma curvesand white point stored in memory of the theater display system, and ifnot in conformance (within a preset tolerance) a color transformation isestablished to correct for the error, as described earlier, such thatthe color output by the theater system is substantially the same as theaim values (step 1891). The goal of the calibration at steps 1890-1891is to make sure that the actual, operating gammas in the theaterdisplay's color channels are such that the gamma and white point“expectations” of the profile are adequately approximated.

When color calibration is verified at step 1891, the program checks tosee if spatial uniformity calibration was requested (step 1892). If not,the program exits, releasing the handset. Otherwise, spatial uniformitycalibration is carried out (step 1893), preferably according toprocedures outlined below, especially FIGS. 40 through 43B. When suchcalibration is complete, the calibration program exits, releasing thehandset which is re-enabled to perform other functions.

The handset is useful as a color calibrator when there is no calibrationperipheral as described earlier (i.e., sensor 1508 or CMI module 1608).If at step 1877 such peripheral were present, then procedures forcalibration with such peripheral are used, as described earlier or step1880-1993 may be performed as described above using the sensor 1508 orCMI module 1608 to provide images of the theatre display rather than thehandset.

In the case where the spectral response functions of the handset cameraare known by virtue of factory or aftermarket calibration, more exactingcalibrations may be possible. Based upon data stored in handset memoryor available from the Calibrator Camera Server. If the spectraltransmission/emission functions of the display-to-be-calibrated are alsoknown, at least to a good approximation, then comprehensive colorcalibration (including profile and multi-dimensional transformgeneration) based on locally measured data are possible, especially withcomputational help from Calibrator Camera Server.

Briefly, the spectral response functions of both the handset camera'scolor channels and the CIE Standard Observer (these are referred to ascmfs or Color Matching Functions) are convolved with various levels andcombinations of activation of the theater display's channels,numerically. In this way, simulated data on the responses of the handsetcamera and of a true colorimeter to colors of the display areaccumulated. These data can be used to develop a heuristictransformation from handset camera measurement to colorimetric values ina procedure similar to the IT8 scanner calibration method referencedearlier. Other methods for conferring at least an approximatelycolorimetric calibration on the handset camera are also compatible.

The foregoing is summarized in FIG. 39J, which is drawn with referenceto processing best carried out at/by the Calibrator Camera Server. Inthis figure, the term “calibrator” refers to the handset camera. First,the spectral response functions of the handset camera to be used forcalibration are read (step 1894). These functions could be retrievedfrom the database of the Calibrator Camera Server, a manufacturer's website or from the specific handset itself, where the curves had beenstored following factory calibration. In like manner, spectraltransmission or emission functions of the display-to-be-calibrated areretrieved (step 1895). Then, various proportions of the display'scolorants are mixed (step 1896). For a display which observes linearrules of color mixture, as discussed earlier, this amounts to scalingeach of the emission curves by various constant factors and addingtogether the weighted curves for the respective color channels.

Next, emission curves so mixed are convolved (see earlier portions ofdisclosure or the referenced CIE Publication No. 15.2) with the HumanColor Matching Functions to produce TriStimulus Values (TSVs) and thenwith the Handset Camera's spectral responsivities to produce,analogously, tsvs, where lower case indicates that they are approximate,not true relative to the Standard Observer (step 1897). Lastly, athree-dimensional space to three-dimensional space transform isdetermined which approximately yield true TSVs as a function of thehandset display's tsvs (step 1898). This may be performed by using aleast squares polynomial fitting or similar procedures to develop amapping of tsvs to TSVs. By such methods, improved measurements of colorproperties, including gamma and white balance, may be provided. Thetransformation determined at step 1898 are stored in memory of the colordisplay system-to-be-calibrated and/or memory of the handset (step1899).

As stated earlier, for the calibration of “Another Displays” (1976 inFIG. 39F), it will be useful to provide a support for the handset so asto relieve the user of responsibility for holding it steadily duringcalibration procedures. The support could be similar in design to thebattery-charging base for a handset and could even be such base whensuitably adapted. FIG. 39K illustrates a possible configuration withrespect to a control unit, such as control unit 1602, where no CMImodule 1608 is present and calibration is desired. In this figure isshown a display screen or surface 1601, and handset 1930 with its camera1943 and display screen 1931, as in FIGS. 39C and 39D. The camera 1943has a field of view 1993 of screen 1601. Eye 1992 represents the eye ofthe user viewing handset display screen 1931 along the dashed line ofsight. The upper housing 1930 b is releasably coupled to support 1991.As such, the support 1991 mechanically latches to the upper housing 1930b. Such latching for example may be by locating the upper housing 1930 bin a recessed molded cavity 1991 a of the support 1991, and one or moreadjustable and/or elastic straps or bands 1991 b from the sides of suchcavity that are adjustable to retain upper housing 1930 b in a stableposition while enabling viewing by user's eye 1992 of display 1931.Latching may also be provided by a combination of interlocking grooves,prongs, Velcro, tabs, or other feature, along the surfaces upon whichthe support and upper housing 1930 b mate. Other configurations arefeasible, the key factors being that the camera 1943 has a stable viewof the screen 1601 and that the user be able to use viewfinder 1931without interfering with data collection by camera 1943. If needed, theentire support 1991 may be movable by the user for aligning camera'sview 1993 with features on screen 1991 during alignment (step 1881 ofFIG. 39I) upon the controller unit or on another surface, such as atable or stand enabling viewing of camera 1943 of screen 1601. AlthoughFIG. 39K shows control unit 1602, optionally control unit 1706 may used,or other projection or panel display systems which are programmed tooperate in conjunction with handset to enable color calibration of theirrespective displays.

Additionally, intelligence may be designed into the system such that,for example, glare hotspots due to specular reflections in the camera'sfield of view are identified and reported. More generally, if diffuse orspecular reflections from the theatre display screen are such as tosignificantly reduce color gamut, the user may be given an advisory thatsystem color capability or accuracy is compromised.

Referring to FIG. 39L, an optional connector 1913 is shown for mountinghandset 1930 to computer 1900 of FIG. 39 in lieu of camera 1910.Connector 1913 has a shaft 1913 a having a right angular bend. The end1913 b of the shaft is threaded, such that it mates with threadedreceptacle or opening 1994 a or 1994 b along lower housing 1930 c. Theother end 1913 c of the shaft 1913 a is coupled to arm member 1906 suchas by friction fit into an hole at the end of shaft 1912 with camera1910 removed. Use of two receptacle 1994 a and 1994 b provides twodifferent placements for mounting handset 1930 in which the handsetcamera operates en lieu of camera 1910 so as to be able to subservesimilar, multiple functionality, as described earlier. Receptacle 1994 aand 1994 b are shown in FIG. 39K for purposes of illustration, and suchhandset 1930 would be removed from support 1991 when coupled viaconnector 1913 to computer 1900. More generally, threaded tip 1913 b maybe compatible with a tripod-like mounting fixture such that a user mayuse a compact digital camera of any sort, rather than handset 1930, ifdesired, for connection to arm member 1906. The handset (or such otherdigital camera) then transmits color images captured from its camera1943 to computer 1900 to enable use in calibration or videoconferencing, similar to camera 1910 described earlier, viacommunication via a cable to serial interface 1961, or wireless viareceiver/transceiver 1956 of FIG. 39E, where the portable computer 1900has an interface or port enabling communication via such interface 1961or receiver/transceiver 1956. The mounting of a handset and/or a digitalcamera to a conventional or notebook computer 1900 is not limited to theuse of shaft 1913 and/or arm member 1906, for example, arm member 1906may be replaced by another shaft, which may be straight, bent, orflexible (e.g. like a snake light shaft) to extend from handset 1930 andcouple to the portable computer's housing.

As shown above, the color of displays of portable handheld device, suchas shown in FIGS. 39C-D, or larger portable devices such as portablelaptop computer, such as shown in FIG. 39, have built-in automatic colorcalibration capability. In each case, the camera is a dual use camerasince it is able to capture images of calibration forms on theirrespective displays, when the either camera (FIG. 39) or display (FIGS.39C-D) are pivoted to enable the camera to view the display, and alsofor use as a typical camera for capturing images of user, objects, orscreen, as the user desires. The camera of the portable handheld device,or the larger portable device of FIG. 39, may each be used as a colormeasuring instrument for calibrating another display once such camera isproperly aligned to view such display. Portable devices with built-incamera other than those illustrated in the figures may also be used byprogramming of their software to enable calibration of the color oftheir display or another display.

Large screen display devices of FIGS. 35, 36, 37, and 37B, or smallscreen display of FIG. 39C, often have non-uniformities of the colordisplayed over their respective screens. Such non-uniformities aredetected and corrected according to the improved method described belowat their respective control units, or by the microprocessor-basedelectronics of such displays when the functionality of the control unitis provided by such electronics. The problem of non-uniformities of thecolor displayed may also occur on CRT display, smaller self-luminousdisplays and other rendering surfaces (such as paper), but may be lessnoticeable to users than larger display screen formats. Measurement andcorrection of such spatial effects was also described earlier.

First, consider the interaction of spatial uniformity correction withthe processing and transmission of image data from a display subsystemor controller of the computer or digital system to the display device:FIG. 40 illustrates a display screen 2000 on which three points, A, Band C are identified. In the vicinity of the foregoing points, themeasured luminance and color may differ, even when all correspond topixels of identical color in the digital representation of the image.Although described in terms of a video display, spatial uniformitycorrection described herein also applies to hard-copy systems (e.g.,printers, copiers, or presses) where, for example, phenomena such as“ink rob” or uneven charging of a photoreceptor or transfer belt mayresult in non-uniform reproduction of a flat-field image in lithographicor xerographic systems, respectively. Further, spatial uniformitycorrection also is applicable to “mosaic”-type displays comprisingmultiple, butted panels or screens upon each of which part of theoverall image is displayed.

In the course of physically rendering digital signals representative ofstored page or image data, rendering devices usually impose a non-linearconversion. In other words, the modulation of light emitted by orreflected from the rendering medium (e.g., display screen or hardcopy)is not a linear function of the signal applied to the rendering device.In the case of a display that employs Cathode Ray Tube (“CRT”)technology, the relationship between light output and voltage applied tothe electron guns that stimulate the colored phosphors to emit light isapproximately a power law. The light emitted is approximately equal tothe square of the applied voltage and the exponent is called “gamma”.For simplicity, spatial uniformity correction is described below forsuch a CRT display.

FIG. 41 illustrates the typical flow of signals through a CRT (orCRT-like) display system which reproduces a homogeneous, flat-fieldinput non-uniformly. In the course of rendering, two transformationsoccur. Light output at a pixel or small neighborhood on the screen,I_(pix), grows approximately as the power (gamma or y) of the appliedvoltage. For simplicity, assume that gamma is uniform across the screen,but that there are regional variations in output that scale linearly. Inother words, the spatial non-uniformity function I_(spatial)(x,y) issuch that the ratio of the intensity measured at two different x,ycoordinates on the screen (such as points A and B in FIG. 40) does notdepend on the level of the uniform drive to the electron guns. Suchspatial non-uniformity function of the display screen can be measured bycolor measuring instruments described earlier.

Additional complexity is added by the fact that image data stored invideo memory is usually some form of coded RGB. Some softwareapplications for creating image data or for apprehending it from animage capture device (e.g., scanner or digital camera) store “calibratedRGB”, in which the image data may be supposed to have an exactinterpretation in terms of CIE colorimetry. Whether the RGB data arecalibrated or not, a typical encoding is the (approximate) inverse ofthe gamma transformation imposed by the physical rendering device. As aresult of the foregoing pre-compensation for the display physics, thedata are often referred to as being in “gamma-corrected” form.

There are two, main reasons for storing gamma-corrected data:

1) Data are represented in a more visually uniform way, so that fewerbits are required for storage. Twenty-four bits (a byte for each of Red,Green and Blue channels) are insufficient for representing linear RGBdata without suffering undesirable (visually noticeable) quantizationartifacts on display. However, the inverse-gamma-correction resemblesthe non-linear transformation of Luminance that has been found to createa perceptually uniform Lightness scale. For example, CIE PsychometricLightness, L*, is approximately a cube-root transformation of cmLuminance or “cap Y”. Psychometric Lightness is a perceptually uniformscale.

2) Pre-compensation of stored RGB data enable them to be transferreddirectly into a very simple frame buffer and displayed satisfactorilywith little additional processing.

Assuming that the encoding of image data in video memory is known, theproblem of rendering color accurately reduces to that of insuring thatdiscrepancies between the encoding of the digital data in memory and theactual conversion function imposed by the device are compensated, andthat correction is made for the spatial non-uniformity function acrossthe display surface, if such be needed or desired. For example, ifstored image data have been “companded” according to an inverse gamma of1/1.8, but the physical device gamma is 2.2, a harmonizing adjustment isneeded to insure accurate display. The LUT 2002 b shown in FIG. 41 mayprovide for such harmonization.

Linear transformations of image data, such as conversion from calibratedRGB for one set of primaries to RGB for a different set (for example,transforming “sRGB” to “Apple RGB”, where the former are two commonencodings) are usually accomplished, correctly, by decoding the RGB to alinear form before applying the re-mixing coefficients. Given ourassumption about the nature of spatial non-uniformity, corrections arebest applied to linear RGB data, but may be applied to other colorchannel formats.

Assume, for simplicity, that the physical conversion imposed by therendering device is an ideal power function, with power=γ. Let P be thelinear signal in one of the color channels and let nu, ν_(i,j), be thespatial non-uniformity correction factor for a pixel, where i,jsubscripts indicate the pixel position in x,y coordinate space on thedisplay surface. ν is always a fraction (except, of course, at thosepixels having I_(min), where ν=1) and is the ratio of the least brightpixel to the pixel-to-be-corrected (expressed as I_(min)/I_(i,j)).

Applying the correction factor to linear P followed by gammacompensation, and then sending the signal through the rendering device'sintrinsic, physical conversion may be described by the followingequation:ν_(i,j) *P=[(ν_(i,j) *P)^(1/γ)]^(γ),where the left side of the equation is the desired scaling result andthe right side the signal processing or data flow just described. Insidethe parentheses, the non-uniformity correction scales the linear colorsignal on a pixel by pixel basis; i.e., ν_(i,j) has subscripts but Pdoes not. Once the pixel-dependent correction factor has been applied,then the brightest pixel (and all other pixels) will have been reducedto the brightness of the dimmest pixel and the display surface will beuniform, at least in terms of brightness.

It was mentioned earlier that it is typical to find gamma-corrected datain video memory, rather than linear RGB. Applying a spatialnon-uniformity correction directly to gamma-corrected data isrepresented, symbolically, as:[ν_(i,j)*(P ^(1/γ))]^(γ),which simplifies to ν_(i,j) ^(γ)*P. Therefore, unless the non-uniformitycorrection factors are themselves gamma-corrected exactly as are thecolor coordinates in video memory, spatial uniformity will not beachieved on output and tonal distortions will be introduced.Gamma-correcting the spatial non-uniformity correction factors (as inν_(i,j)′=ν_(i,j) ^(1/γ)) are preferably determined prior to devicerendering of user image data, such as in an “off-line” process (e.g.,such as calibration setup of a rendering device performed wheninstalled, and then periodically thereafter or as desired by the user).Although spatial non-uniformity correction may be provided at thefactory, it is preferred that it is performed at the user's location.

Returning now to the FIG. 41: Spatial uniformity correction 2002 a maybe applied to image data accessed from video memory 2001, as indicatedby the dashed line labeled ν(x,y). Either the image data should bedecoded prior to the application of the non-uniformity correction or thecorrection factors should be gamma-corrected exactly as are the imagedata. For example, spatial uniformity correction 2002 a may be providedby a table, such as two-dimensional look-up-table for each pixelcoordinate in an image to be rendered. Alternatively, the image datathemselves may be subjected to a non-uniformity- and gamma-correctionprior to being written to a frame buffer or image store for display. Alook-up-table (“LUT”) 2002 b is provided which can store factors thatcorrect for departures of the physical display's transfer function fromthat of an ideal power function. The latter may be due to intrinsicproperties of the display or, to some degree, to influences of ambientillumination. However, the ability to correct for the latter in this waymay be limited. In any case, the contents of the LUT 2002 b shouldgenerally be based on measurements of the “Voltage-Intensity transferfunction” of the rendering device.

As will be described below, spatial uniformity correction table 2002 aand LUT 2002 b has for each pixel coordinate the correction for each ofthe color channels for the pixel. For example, at pixel (1,1) thespatial uniformity correction table 2002 a for color channels R, B, andG may hold values 0.85, 0.95, and 0.92, respectively, while at pixel(1,2) the values may be 0.86, 0.94, and 0.92, respectively, and so forthfor the entire two-dimensional display screen. Unlike spatial uniformitycorrection table 2002 a, LUT 2002 b is not dependent on pixel location.One of the most common applications of LUT 2002 b is compensation fordifferences between the assumed transfer characteristics of the physicaldevice (that is represented in the encoding of the image data) and theactual transfer characteristic which is usually better calibrated at theuser site than at the factory.

Thus, during rendering device operation in each color channel pixels ofthe two-dimensional (x,y) image data accessed from memory 2001 arecorrected in their value by spatial non-uniformity look-up-table 2002 a,and then by LUT 2002 b. The resulting two-dimensional (x,y) color imageis converted into analog signals by the digital-to-analog converter(DAC) 2003 for output to display 2004. The determination of a spatialnon-uniformity look-up-table 2002 a in multiple color channels isdescribed below.

Interactions of three or more color channels need to be accounted for inspatial non-uniformity correction. The concepts advanced in theforegoing apply to each channel individually, except that thecalculation of the correction factors should be done in such a way as tobe consistent with a targeted neutral (or white) balance. Given thatspatial non-uniformity correction reduces the dynamic range (“contrast”)and resolution (the number of displayable digital codes) of the monitor,it may be desirable to hold such effects to a minimum by calculatingdifferent non-uniformity corrections for different white points. Inother words, for each neutral balance desired, a different spatialnon-uniformity look-up-table 2002 a is determined and stored in memory.Also, such different tables 2002 a may be selected by the user (orautomatically) in accordance with different ambient light situations. Ina multi-channel correction, the value of I_(min) used in calculating thecorrection factors is the least pixel response to a flat field digitalinput from all those across the display surface and across all colorchannels. Re-stated, it is the weakest channel response at the dimmestpixel.

Note that, in the course of performing spatial uniformity correction,there is a possibility of altering the white point or neutral balanceaway from the desired value. Indeed, the display surface is likelynon-uniformly bright and non-uniformly white at the outset. Where linearrules of color mixture and superposition apply, this means that ifneutral balance were the same at every pixel, then the pixel at whichI_(min) occurs would be the same in all three channels at the outset.White balance, in particular, is discussed in connection with the firstequation presented in U.S. patent application Ser. No. 09/832,553, whichis incorporated herein by reference. The essential concepts are reviewedbelow.

The TriStimulus Values (TSVs) X, Y and Z, can be thought of as theamounts of three lights (“additive primaries”) which need to be mixed oradded together in order to match another light of arbitrary wavelengthcomposition for the Standard Observer. Consider the following set ofsimultaneous equations, in which we use linear combinations of the TSVsto “match” (or to provide a device independent numerical specificationfor) the red, green and blue lights emitted by a monitor:R=a ₁₁ X+a ₁₂ Y+a ₁₃ ZG a ₂₁ X+a ₂₂ Y+a ₂₃ ZB=a ₃₁ X+a ₃₂ Y+a ₃₃ Z

Normally, a display is set up, possibly at the factory or before orduring calibration, so that full scale drive to R, G and B channelscorresponds to a particular “white”, such as Daylight 6500 (“D65”) or acolor temperature of 9300° Kelvin, etc. (The two colors just mentionedare both considered “white” but have very different TSVs.) The set up isoften effected by adjustments to variables such as offset and gain inthe control circuitry of the display. The result, ideally, is aparticular relationship, R:G:B, in the light output of Red, Green andBlue channels. In order for a display screen to be truly uniform incolor, not only must the overall intensity at each pixel be constant,but the ratios of R, G and B light output must also be constant. Thismay require adjustment (possibly iterative) in a field-flatteningprocedure such as that described below.

Without limiting the invention, consider, for clarity and simplicity ofexplanation, the determination and use of I_(min) in the case of a CRTdisplay, which approximates the model described above: to determineL_(min) the screen is flooded, successively, with a relatively highlevel of each color, R, G and B. Locate the pixel, in x,y coordinates onthe display surface, having the smallest fraction of the value of thebrightest pixel. As noted, the x,y coordinates of the minima for R, Gand B may not all be the same, indicating non-uniform neutral balance.

Let I_(min) for the Red channel be called R_(min) and let it occur atpoint A of FIG. 40; likewise define G_(min) and B_(min) and let themoccur at points B and C, respectively, of FIG. 40. Suppose, for example,that R_(min)=0.85, G_(min)=0.94 and B_(min)=0.93. The fractions employedto “flatten” (make uniform) the G and B channels then, should be basedupon G and B values that will yield the correct white balance at the x,ycoordinate of R_(min) (point A), unless either or both of thosecalculated results exceed G_(min) or B_(min) values observed initially(at points B and C). In the latter case, at least points B and C (andpossibly many others on the display surface) can not be brought to thecorrect neutral balance without modification to the Red channel; thelatter situation may not arise if the display surface is close to thedesired white balance to begin with and/or spatial non-uniformities arenot too great. If, however, the situation does arise, then the otherchannels are examined to find the minimum consistent with the desiredwhite balance. That failing, the pixel is found that is as close aspossible to the desired balance and then scale down the fractions in R,G and B channels, for that pixel, until the fractions are such that allpixels are correctable. In summary, once a minimum has been found in atleast one of the channels such that all channels can be adjusted toachieve neutral balance, then all pixels can be scaled in each channelaccording to R, G and B values at that minimum pixel. A verification ofuniformity and white balance may be made once the spatial non-uniformitycorrection table has been prepared and applied. The detection anddetermination of the spatial non-uniformity correction table isdescribed in more detail later in FIGS. 43A and 43B.

Referring to FIG. 42, a system 2006 providing multi-channel spatialnon-uniformity correction is shown. System 2006 has sensor 2008 asdescribed earlier, such as an imaging sensor (CCD), a rendering device2009 for outputting images onto a display surface 2010, and aprocessor(s)/controller(s) 2012 which provide image data to therendering device and receive measurement data from sensor 2008. Theprocessor(s)/controller(s) may represent a microprocessor-based device,such as a personal or desktop computer system, or the control units 1602or 1706 described earlier, or a specialized display processor/controllerwithin the host or directly with a controller within the display itself.Sensor 2008 may represent the CMI module 1608 of FIG. 37 or 37B, orsensor 1508 of FIGS. 35 and 36, or the sensor may be a “dumb” sensorcontrolled externally through an I/O port. The sensor 2008 may have amicrocontroller that communicates with the host (general-purpose,programmable) computer providing processor(s)/controller(s) 2012.

The processor/controller operates in accordance with a software programin its memory which may be executed by one or more of the processors andat least causes test images to be displayed by rendering device 2009 ondisplay surface 2010, causes the displayed/rendered images to be sensedutilizing sensor 2008, and analyzes the sensed image data for thepurpose of creating in memory a spatial non-uniformity correction table.In reference to FIG. 41, the spatial non-uniformity correction table2002 a, image store 2001, LUT 2002 b are provided for in memory ofprocessor(s)/controllers 2012, and the processor(s)/controllers 2012outputting via DAC 2002 images to rendering device 2009, e.g., display2004.

Referring to FIGS. 43A and 43B, a flow chart is shown for the programcarried out by processor(s)/controllers 2012 for spatial non-uniformitycorrection. For each color channel j, where j=1 . . . m, a flat-field ofcolor j is rendered 2014 on display surface 2010 by rendering device2009 (e.g., each pixel is at or near its highest pixel value for thechannel over the display surface). Sensor 2008, being earlier alignedand focused on the display surface, measures color as a function ofspatial position (x,y) on the display surface 2010 (step 2015). Suchmeasurement for each pixel of the screen is stored in memory of theprocessor(s)/controllers 2012. The coordinate (x,y) of the pixel on thedisplay surface having the lowest fraction I_(min)/I_(max), isdetermined (step 2016) and saved in memory of processor(s)/controllers2012. I_(min) represents the least pixel response to a flat fielddigital input from all those across the display surface for colorchannel j. I_(max) represents the most pixel response to a flat fielddigital input from all those across the display surface for the j^(th)channel. In other words, for each channel j=1 . . . m, I_(min)/I_(max)is the ratio of the intensity of the dimmest pixel to the brightestpixel, and pixel coordinate (x,y) with the lowest I_(min) is saved inmemory.

After steps 2014, 2015, and 2016 are performed for each color channel(step 2018), the channel j with the smallest saved fractionI_(min)/I_(max) is identified, referred to herein as channel j_(min)(step 2020). For the pixel (x,y) having I_(min) in channel j_(min), theintensity I in the fraction I/I_(max) for all other channels thanj_(min) are determined such that pixel (x,y) has the desired neutral orwhite balance (step 2022), where I_(max) for the channel was determinedat step 2016. In other words, the brightness of the pixel (x,y) in theother channels than channel j_(min) is modified, if necessary, in orderto satisfy a proportional relationship between the color channelsdefined by the neutral color balance preselected by the user. Theresulting I/I_(max) in each of the other color channels than j_(min) isthen compared to the saved I_(min)/I_(max) for the respective channelfrom step 2016. If to obtain the desired neutral balance of pixel (x,y),the fraction I/I_(max) for that pixel (x,y) determined at step 2020 ineach of the other channels is less than or equal to fractionI_(min)/I_(max) of that channel determined at step 2016 (step 2023),then the spatial non-uniformity correction table is prepared and stored(step 2024), otherwise the process proceeds to step 2026 of FIG. 43B.

For step 2023, to prepare the spatial non-uniformity correction LUT atstep 2024 is as follows: the spatial non-uniformity correction ν forpixel (x,y) in color channel j_(min) is set to one, and for each otherpixel ν is set to the fraction that makes its intensity at step 2016equal to the measured intensity of pixel (x,y). In each of the otherchannels than channel j_(min), the spatial non-uniformity correction thepixel (x,y) has ν set to fraction I/I_(max) for the channel determinedat step 2022, and each other pixel has ν set to the fraction that makesits measured intensity at step 2016 equal to the measured intensity ofpixel (x,y) for the channel when reduced by fraction I/I_(max) for thechannel.

If to obtain the desired neutral balance of pixel (x,y), the fractionI/I_(max) for that pixel (x,y) determined at step 2022 in the otherchannels is greater than the fraction I_(min)/I_(max) of that pixel(x,y) determined at step 2016 (step 2023), then branch to step 2026 ofFIG. 43B. The pixel (a,b) is identified in the other channels thanchannel j_(min) having lowest saved fraction I_(min)/I_(max) from step2016 (such channel is referred to herein as channel n_(min)), and forthat pixel (a,b) one or more fractions I/I_(max) are determined in eachof the other color channels than n_(min) to obtain the desired neutralbalance. The computation at step 2026, as at step 2022, employed thecalibration matrix for the display to calculate I values that ensure thecorrect proportions of R:B:G. The calibration matrix is that used forinterconverting R,B,G codes for a particular set of colorants tophosphors and device independent units, such as XYZ TriStimulus values.One or more sets of fractions may be determined to achieved the desiredneutral balance. If any set of fractions ensured uniform neutral balance(step 2028), than the best set is picked (step 2029), and thenon-uniform correction table is prepared at step 2024. Step 2028 isperformed the same as step 2023, where the calibration matrix is used tomake sure that a minimum in one channel can ensure neutral balance inall three channels. For example, the best set may be the one which onaverage reduces the least the intensity of all the color channels forpixel (a,b). From step 2029, the spatial non-uniformity correction tableat step 2024 is prepared as described above, except that the channelwhose I_(min)/I_(max) is used has changed in accord with steps 2026 to2028.

If no set of fractions at step 2028 provides a uniform neutral balance,then one pixel is found on the display having the correct neutralbalance on the basis of measurements (which may need to be interpretedin light of the calibration matrix) (step 2030). If no such pixel isfound, a pixel is created with the correct neutral balance by startingwith a pixel having approximately the correct balance and using thecalibration matrix to calculate the correct ratios. At this “created”pixel, the fraction I/I_(max) are reduced for all channels, consistentwith neutral balance, until the I_(min) in each of channel is the globalI_(min) over the display surface for that channel. When complete, thespatial non-uniformity correction table at step 2024 is prepared, inwhich the spatial non-uniformity correction table has ν set to thefractions I_(min)/I_(i,j) determined on the basis of results at step2032 for each pixel in each channel. The term “pixel” in the foregoingdiscussion may be an average of a small group of pixels and the methodis able to exclude “dead” pixels or other outliers from consideration.In addition, the less-than-or-equal-to comparison at step 2023 caninclude a tolerance for some error, so that the computation does not get“bogged down”.

The spatial non-uniformity correction table determined at step 2024 maybe incorporated into the color transformations described earlier for arendering device, and may also be useful in the case of renderingdevices whose color mixture is non-linear. Also, the spatialnon-uniformity correction table may be used in conjunction with thehistogram method for identifying color errors of image reproductionindependently of image resolution described earlier (see FIG. 20 andattendant discussion). Used alone, a determined histogram may bevulnerable depending on the display device to non-uniformities of coloracross a display surface which may give rise to ambiguities ofinterpretation of color error data. In other words, are there blurred,multiple or misplaced peaks in a measured histogram due to spatialnon-uniformities of the display? Field-flattening described herein maybe used either to correct the display surface prior to application ofverification procedures that test for temporal drift in the renderingprocess, such as the histogram method. Field-flattening may also beemployed to pre-process image data apprehended from a non-uniformrendering surface, so that resolution-independent methods of detectingtemporal drift in color reproduction can be applied unambiguously.Alternatively, the reference histogram may be modified to reflect thespatial non-uniformities of the actual rendering process prior tocross-correlation.

Please consider, now, some additional features and advantages of themethod: I_(min) and the fractions stored in the spatial non-uniformitycorrection table may be computed with attention to allowing “headroom”or some degree of over-correction. The latter would permit changes towhite balance (up to a point) without necessity of re-computing thenon-uniformity correction. In the example of the CRT display devicedescribed above, it would generally be advantageous to make the digital,input, uniform-field images of relatively high brightness. However, inapplication of the method to a subtractive, reflection,color-reproduction medium (such as print) it will be best to make theinput image one of moderate density. Indeed, for such devices havingmore complicated color-mixture physics, highly accurate correction ofthe display surface may require measuring flatness at several levels ofdensity and computing correspondingly more complicated correctionfunctions.

In summary, once I_(min) is measured, preferably in some standardphotometric or colorimetric coordinate system (such as CIE), then thevalues for the other channels should be chosen so as to achieve thedesired white point, or balance between the channels. LUT 2002 b of FIG.41 for compensating peculiarities of the physical device may comprise aseparate table for each channel. In the case of a hard copy device, theseparability of “gamma” (i.e., for the non-linear relationship betweendigital drive and some measure of physical output) from the processesgoverning color mixture cannot, in general, correctly be assumed. Inthis case, sufficiently accurate non-uniformity correction may requirethat the LUT 2002 b be a three-dimensional table for each color channel.Two of the address coordinates for each table represent spatialposition, x,y, on the page or display surface. The third, of course, isthe value of the color coordinate in that neighborhood. Methods forcalculating the entries of such a table are extensions of methods forcalculating the colorant values needed to realize a desired color in thereproduction that are described earlier.

For example, in the case of a CRT-type video display, the spatialnon-uniformity correction functions may be fairly simple and invariantover time, provided that the magnetic fields impinging upon the displayare consistent. In the case of an offset press capable of printing manydifferent pages (of a publication, for example) in a single imposition,the situation is more complex and likely to vary from job to job.Currently, the exact color reproduced on a given page of the signaturemay depend markedly on where the page is situated within the signaturedue to phenomena such as “ink rob” mentioned earlier. The term “ink rob”refers to the relative print densities achievable at a trailing locationof an impression as a function of reproduction densities required at amore leading position within the same impression (transfer of ink topaper). Today, press operators try to cope with regional dependencieswithin the press sheet by seeking an average optimum or by favoringparticular advertisements within the signature.

Conceptually, methods disclosed herein enable more customized colorcontrol within a complex signature. By analyzing the color reproductionperformance of a press in response to a variety of inputs and developinga data base of spatial dependencies, a model of press performance iscreated. The model informs processing of the pre-press workflow,especially at the stage where pages are imposed or laid out in asignature. Operators are given a more realistic preview of colorreproduction to be expected from a given layout and press run.Additionally, they may be given a tool enabling the calculation of oneor more spatial uniformity correction functions to be applied within thesignature, or, alternatively, may instruct the system to computeregionally customized (for different pages within the signature)transformations from color to colorant in the interest of more accuratecolor reproduction. To the extent that complete compensation for thespatial non-uniformities of the reproduction process is not possiblewithin a particular imposition, operators have the option ofexperimenting with different impositions which may cure the deficiencyor, at the least, of informing the print buyer regarding what colortolerances can be satisfied. The methods disclosed herein are alsoapplicable to “mosaic” video displays in which one large screen isactually comprised of an array of smaller, discrete displays.

Referring to FIG. 44, the minimum apparatus needed to support theforegoing method and functionality is a production press 2034, a colormeasurement instrument 2036 (preferably an imaging colorimeter), one ormore computers 2038 having memory for database storage 2040 and anetwork interface 2042 supporting communication with sites that performimposition. At least part of the network communication may be wireless,to the extent that bandwidth is adequate. Minimally, the computers 2038have a program to enable reading and analyzing color measurement dataprovided from color measurement instrument 2036 and refines or updatesdata and models of position-dependency of color reproduction withinprinted signatures that are stored in the database. The process isdescribed in FIG. 45. First, color image data is captured from pageswithin signature (step 2044). Next, the model of positional dependencesof color on press sheets is refined and revised (step 2046). Thereafter,one or more of the following at steps 2047 a,b, or c is performed.Customized spatial non-uniformity correction table is prepared forselected page(s) at one or more candidate positions within the signature(step 2047 a). Customized rendering transform for device for selectedpage(s)s at one more candidate position with the signature (step 2047b). Customized color-to-color' transformation for proofing of selectedpage(s) at one or more candidate position within the signature (step2047 c). The pre-press operator may select one or more of steps 2047a,b,c to be performed. The resulting table and/or transformations aremade available to the one or more nodes in the network to which thecomputers 2038 are in communication with via network interface 2042(step 2048).

Based upon foregoing data and models, customized non-uniformitycorrections or rendering tables may be prepared on instruction frompre-press, most likely the imposition stage of production. Within theframework of Virtual Proofing described earlier, non-uniformitycorrection tables and rendering tables normally are stored locally.Information shared with node(s) upstream in production would be thecolor-to-color' mappings needed to represent imagery as it will berendered in volume production along with associated information aboutpage placements within the signature. The imposition stage may thenrelay the proofing representations to others, such as the print buyer orher/his agent, when satisfied with the anticipated results. Theforegoing is outlined in FIG. 46. The press operator workstation 2049(e.g., computer 2038 having database 2040) sends transforms 2050 forcolor image data incorporating position dependencies of colorreproduction to a pre-press imposition workstation 2052. Workstation2049 is coupled to a press 2034 (FIG. 44), such as a web offset press.Workstation 2052 may represent a computer 2052 with a network interface2052 a for receiving the transforms 2050 and may apply the transforms toimage data representing page(s) for output to its display 2052 b or toan optional hardcopy output device (e.g., printer) 2052 c. At thepre-press operator workstation an election is made as to positions ofcolor critical data within the press sheet(s), and such is communicatedto the press operator workstation. Such election 2054 may occur prior tothe generation of transforms 2050 at press operator workstation; or, ifmade later, such transforms 2050 may be revised at press operatorworkstation. One or more other workstations 2056 provide proofinginformation (virtual proof) and client approval 2057. One ofworkstations 2056 is illustrated, and may represent a computer with anetwork interface 2056 a for receiving the transforms 2050 which may beapplied to image data representing page(s) for output to its display2056 b or to an optional hardcopy output device (e.g., printer) 2056 c.As such workstations 2038, 2052, and 2056 each may represent a node orsite along the network of FIG. 1. It may also be expedient to share thenon-uniformity corrections and/or rendering tables with sites/nodesupstream in production that are capable of computing proofing transforms(for their associated rendering devices) therefrom.

Returning, now, to the application of the spatial non-uniformitycorrection: The granularity (relative coarseness of spatial sampling) ofthe non-uniformity correction tables is a design choice reflecting atrade-off between circuit and computational complexity (often embodiedin a display processor) and amount of available memory. In U.S. Pat. No.5,510,851, a spatial interpolation method for applying non-uniformitycorrections based upon Catmull-Rom splines is preferred. The spatialnon-uniformity correction described herein is compatible with varioustrade-offs between memory and computational and hardware complexity aswell as with various methods for interpolating and/or smoothing appliednon-uniformity corrections.

From the foregoing description, it will be apparent that there has beenprovided a system, method, and apparatus for color calibration.Variations and modifications in accordance with the invention willundoubtedly suggest themselves to those skilled in the art. Accordingly,the foregoing description should be taken as illustrative and not in alimiting sense.

What is claimed is:
 1. A conferencing system comprising: a computerserver; one or more first sites each having a rendering device with acolor display and being associated with a programmable computer system,in which said rendering device comprises: at least one camera; one ormore speakers; a network interface enabling access to said computerserver; and one or more programs that operate the computer systemassociated with the rendering device and enable a user to assemble datacommunicated from said computer server upon said color display of therendering device, said data comprising one or more of graphics, images,text, and numerical data that are a subject of an interactive conferencealong with annotations associated with said data by one or more usersparticipating in said conference; one or more second sites each having ahand-held device comprising: a programmable computer system; a pluralityof sensors linked to said computer system of the hand-held device,wherein one or more of said sensors comprise a camera enabling captureof images, in which said images captured represent different fields ofview; a color display operable by said computer system of the hand-helddevice to present menus of a user interface, to display one or more ofsaid images captured, and to display at least part of said dataaccessible from said computer server in said conference with at leastsaid one or more of said first sites, wherein when a user faces a screenof the color display of the hand-held device said different fields ofview comprise at least a first one of said fields of view directed tothe user and a second one of said fields of view directed away from theuser; one or more microphones and one or more speakers enabling audioinput and output; a transmitter and receiver for enabling cellulartelephony; and one or more programs that operate the computer system ofthe hand-held device and enable at least communication with saidcomputer server using one or more network protocols; and said computerserver having one or more programs for storing said data, andinformation of location or identity of said first and second sites, andenabling communication with the computer system at each of said one ormore first sites, and the computer system at each of said one or moresecond sites.
 2. The conferencing system according to claim 1 whereinsaid rendering device utilizes said hand-held device at one of said oneor more second sites as a peripheral device.
 3. The conferencing systemaccording to claim 1 wherein said one or more programs that operate thecomputer system of said hand-held device further enable communicationwith said computer server to transmit one or more images comprising saidfirst one or said second one of said fields of view to one or more otherusers in said conference.
 4. The conferencing system according to claim1 wherein at least said images or said graphics of said data displayedon the displays of said first and said second sites are stored incalibrated RGB coordinates, and said one or more programs operable atone or more of said computer system of said rendering device and saidcomputer system of said hand-held device further comprise one or moreprograms utilizing one or more calibrated color transformations toconvert said data stored in said calibrated RGB coordinates to digitalcodes for the color display of said rendering device or said hand-helddevice, respectively.
 5. The conferencing system according to claim 1wherein said one or more programs that operate said computer server orsaid computer system of at least said hand-held device further enablevoice and speech recognition and spoken instructions by said hand-helddevice to a user for performing a task.
 6. The conferencing systemaccording to claim 1 wherein said one or more programs that operate saidcomputer server or said computer system of at least said hand-helddevice further enable computer generation of spoken assistance to auser, including a least conversion of documentary information intoaudible instructions.
 7. The conferencing system according to claim 1wherein said one or more programs that operate at one or more of saidcomputer system of said rendering device, said computer server, and saidcomputer system of said hand-held device detect features in spatialpatterns displayed in images on the color display of said renderingdevice or said hand-held device, respectively, and compares saidfeatures to stored geometric shapes.
 8. The conferencing systemaccording to claim 1 wherein said one or more programs that operate saidcomputer system of the rendering device further enable a virtual pencilused with the color display of the rendering device to associate saidannotations with elements of said displayed data, wherein saidannotations are shared with other sites interactively.
 9. Theconferencing system according to claim 1 wherein the data assembled hasresults of processing at least said graphics and said images that arethe subject of the interactive conference along with said annotationsassociated with said data, and at least said graphics are displayed onat least the color display of said rendering device as an a separableoverlay upon one of the images of the data image.
 10. A non-transitory,computer-readable medium encoding software executable by one or morecomputers for conferencing, said software comprising: one or moreprograms operable on a first computer system at each of one or morefirst sites, said first computer system being associated with arendering device having a color display, said one or more programsoperable on said first computer enable a user to assemble datacommunicated from a computer server upon said color display, and utilizeat least one camera, one or more speakers, and a network interface,which are associated with the rendering device, wherein said networkinterface enables access to said computer server, and said datacomprises one or more of graphics, images, text, and numerical data thatare a subject of an interactive conference along with annotationsassociated with said data by one or more users participating in saidconference; one or more programs operable on a second computer system ofa hand-held device at each of one or more second sites, in which saidhand-held device comprises a plurality of sensors, a color display, oneor more microphones, one or more speakers enabling audio input andoutput, and a transmitter and receiver for enabling cellular telephony,said one or more programs operable on said second computer systemutilize one or more of said plurality of sensors which comprise a camerafor capturing images, in which said images captured represent differentfields of view, wherein when a user faces a screen of the color displayof the hand-held device said different fields of view comprise at leasta first one of said fields of view directed to the user and a second oneof said fields of view directed away from the user, and utilize saidcolor display of the hand-held device for presenting menus of a userinterface, displaying one or more of said images captured, anddisplaying at least part of said data accessible from said computerserver in said conference with at least said one or more of said firstsites, wherein said one or more programs operable on said secondcomputer system further enable communication with the computer serverusing one or more network protocols; and one or more programs operableon said computer server storing said data and information of location oridentity of said first and second sites, and for enabling communicationwith said first computer system at each of said one or more first sites,and said second computer system at each of said one or more secondsites.
 11. The non-transitory, computer-readable medium according toclaim 10 wherein said one or more programs operable on said firstcomputer system utilize said hand-held device at one of said one or moresecond sites as a peripheral device.
 12. The non-transitory,computer-readable medium according to claim 10 wherein said one or moreprograms operable on said second computer system further enablecommunication with said computer server to transmit one or more imagescomprising said first one or said second one of said fields of view toone or more other users in said conference.
 13. The non-transitory,computer-readable medium according to claim 10 wherein at least saidimages or said graphics of said data displayed on the displays of saidfirst and said second sites are stored in calibrated RGB coordinates,and said one or more programs operable at one or more of said first andsecond computer systems further comprise one or more programs utilizingone or more calibrated color transformations to convert said data storedin said calibrated RGB coordinates to digital codes for the colordisplay of said rendering device or said hand-held device, respectively.14. The non-transitory, computer-readable medium according to claim 10wherein said one or more programs operable on said computer server or atleast said second computer system further enable voice and speechrecognition and spoken instructions by said hand-held device to a userfor performing a task.
 15. The non-transitory, computer-readable mediumaccording to claim 10 wherein said one or more programs operable on saidcomputer server or at least said second computer system further enablecomputer generation of spoken assistance to a user, including a leastconversion of documentary information into audible instructions.
 16. Thenon-transitory, computer-readable medium according to claim 10 whereinsaid one or more programs operable at one or more of said computerserver and said first and second computer systems detect features inspatial patterns displayed in images on the color display of saidrendering device or said hand-held device, respectively, and comparessaid features to stored geometric shapes.
 17. The non-transitory,computer-readable medium according to claim 10 wherein said one or moreprograms that operate said first computer system further enable avirtual pencil used with the color display of the rendering device toassociate said annotations with elements of said displayed data, whereinsaid annotations are shared with other sites interactively.
 18. Thenon-transitory, computer-readable medium according to claim 10 whereinthe data assembled has results of processing at least said graphics andsaid images that are the subject of the interactive conference alongwith said annotations associated with said data, and at least saidgraphics are displayed on at least the color display of said renderingdevice as an a separable overlay upon one of the images of the dataimage.